CN117294833A - Camera chip testing method and related equipment - Google Patents
Camera chip testing method and related equipment Download PDFInfo
- Publication number
- CN117294833A CN117294833A CN202311316496.9A CN202311316496A CN117294833A CN 117294833 A CN117294833 A CN 117294833A CN 202311316496 A CN202311316496 A CN 202311316496A CN 117294833 A CN117294833 A CN 117294833A
- Authority
- CN
- China
- Prior art keywords
- camera
- chip
- detected
- cheating
- test result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 300
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 59
- 238000000034 method Methods 0.000 claims abstract description 57
- 230000003993 interaction Effects 0.000 claims abstract description 18
- 238000001514 detection method Methods 0.000 claims description 48
- 238000010998 test method Methods 0.000 claims description 11
- 238000007689 inspection Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 abstract description 8
- 238000012544 monitoring process Methods 0.000 abstract description 7
- 238000005516 engineering process Methods 0.000 abstract description 6
- 230000005540 biological transmission Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 8
- 238000013473 artificial intelligence Methods 0.000 description 6
- 230000009191 jumping Effects 0.000 description 6
- 125000000205 L-threonino group Chemical group [H]OC(=O)[C@@]([H])(N([H])[*])[C@](C([H])([H])[H])([H])O[H] 0.000 description 4
- 230000006978 adaptation Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013515 script Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000011835 investigation Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000011056 performance test Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/002—Diagnosis, testing or measuring for television systems or their details for television cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/08—Protocols for interworking; Protocol conversion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
The disclosure provides a method for testing a camera chip and related equipment, and relates to the technical field of intelligent monitoring. Transmitting a connection protocol according to a unified message interaction protocol, and connecting with a plurality of cameras to be detected; issuing chip test data to each camera to be detected so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model; and receiving test result information of the chips returned by each camera to be detected, and determining a chip test result table. The method and the device can be compatible with chip architectures and interfaces of different manufacturers and shield the influence of algorithms on the performance of the chip by combining the unified message interaction protocol with the unified setting algorithm model, overcome the problem of non-uniform test calculation standard of the camera chip in the related technology, and realize the transverse comparison of the chip performance.
Description
Technical Field
The disclosure relates to the technical field of intelligent monitoring, in particular to a method for testing a camera chip and related equipment.
Background
With the rapid development of chips, deep learning algorithms, big data and cloud computing, artificial intelligence technology has been gradually put into use in many industries. With the integration of artificial intelligence (AI, artificial Intelligence), AI intelligent cameras are widely applied in China, have many users and large markets, and can see a spot. At present, the AI front-end camera already occupies nearly 70% of the market of intelligent application of video monitoring, and presents attractive prospect. In the next twenty years, the market total of the AI front-end camera is likely to break through 10000 billions. The core of the AI intelligent monitoring camera is an AI intelligent algorithm, and the main functions of the AI intelligent monitoring camera comprise: pattern classification, pattern recognition, AI movement track tracking shooting and the like. However, the quality of the AI camera is largely dependent on the computing power of the AI chip and the accuracy of pattern recognition. The computational power performance of thousands of AI camera products on the market is uneven.
In the prior art, the product capability of AI camera products in the industry is different, and each manufacturer has interpretation of own product performance. The manufacturer marks the performance index of the product, the calculation standard is not uniform, and the product has no reference value of transverse comparison.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The disclosure provides a method and related equipment for testing a camera chip, which at least overcome the problem of non-uniform test calculation standard of the camera chip in the related technology to a certain extent.
Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.
According to one aspect of the present disclosure, there is provided a method for testing a camera chip, including: according to the unified message interaction protocol, a connection protocol is sent to be connected with a plurality of cameras to be detected; issuing chip test data to each camera to be detected so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model; and receiving test result information of the chips returned by each camera to be detected, and determining a chip test result table.
In some embodiments, the method further comprises: if the chip test result of one camera to be detected and the preset chip performance value meet the preset conditions, judging that the one camera to be detected is a cheating camera to be detected; and performing anti-cheating detection on the to-be-detected cheating camera in a preset anti-cheating mode, and determining an anti-cheating detection result of the to-be-detected cheating camera.
In some embodiments, the method further comprises: calculating a first offset value between the chip test result and a preset chip performance value; and if the first offset value exceeds a preset threshold value, judging that a preset condition is met.
In some embodiments, the preset anti-cheating manner includes: and modifying an algorithm model in the to-be-detected cheating camera chip and/or modifying chip test data.
In some embodiments, when the preset anti-cheating mode includes modifying an algorithm model in the to-be-detected cheating camera chip, the anti-cheating detection is performed on the to-be-detected cheating camera through the preset anti-cheating mode, and determining the anti-cheating detection result of the to-be-detected cheating camera includes: increasing the number of layers of the algorithm model in the to-be-detected cheating camera chip from a first value to a second value, and/or increasing parameter information of the algorithm model from a third value to a fourth value, and testing the chip test data again to determine first inspection test result information; if the second offset value of the first test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
In some embodiments, when the preset anti-cheating mode includes modifying chip test data, the anti-cheating detection is performed on the to-be-detected cheating camera through the preset anti-cheating mode, and determining the anti-cheating detection result of the to-be-detected cheating camera includes: modifying data in the chip test data according to a preset proportion; issuing modified chip test data to the cheating camera to be detected, and determining second inspection test result information; if the third offset value of the second test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
In some embodiments, before the issuing chip test data to each camera to be inspected, the method further comprises: polling to acquire the state of the camera to be detected; when the cameras to be detected are in an idle state, chip test data are issued to each camera to be detected; and when the camera to be detected is in a non-idle state, acquiring the state of the camera to be detected again after a preset time interval.
In some embodiments, before the sending the preset connection protocol connects to the plurality of cameras to be detected, the method further includes: retrieving cameras of a preset Internet Protocol (IP) network segment; and adding the retrieved camera to the equipment list, and determining the camera to be detected.
In some embodiments, the sending a connection protocol according to the unified message interaction protocol, and connecting with the plurality of cameras to be detected includes: sending a hello message; when receiving the information message returned by the camera to be detected, the camera to be detected is connected with the camera to be detected.
In some embodiments, the issuing chip test data to each camera to be tested includes: transmitting a test request message, wherein the test request message comprises chip test data; and receiving a return message after the camera to be detected confirms the request.
In some embodiments, the receiving the test result information of the chip returned by each camera to be detected, and determining the chip test result table includes: sending and obtaining a test result message; obtaining a test result message, wherein the test result message comprises test result information; and determining a chip test result table according to the plurality of test result messages.
According to another aspect of the present disclosure, there is also provided a test apparatus for a camera chip, including: the sending connection protocol module is used for sending a connection protocol according to the unified message interaction protocol and connecting with a plurality of cameras to be detected; the chip test data module is used for issuing chip test data to each camera to be detected so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model; the chip test set determining module is used for receiving the test result information of the chips returned by each camera to be detected and determining a chip test result table.
According to another aspect of the present disclosure, there is also provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of testing a camera chip of any one of the above via execution of the executable instructions.
According to another aspect of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of testing a camera chip of any one of the above.
According to another aspect of the present disclosure, there is also provided a computer program product including a computer program which, when executed by a processor, implements the method of testing a camera chip of any one of the above.
According to the method for testing the camera chip, which is provided by the embodiment of the disclosure, a connection protocol is sent according to a unified message interaction protocol to be connected with a plurality of cameras to be detected; issuing chip test data to each camera to be detected so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model; and receiving test result information of the chips returned by each camera to be detected, and determining a chip test result table. The method and the device can be compatible with chip architectures and interfaces of different manufacturers and shield the influence of algorithms on the performance of the chip by combining the unified message interaction protocol with the unified setting algorithm model, overcome the problem of non-uniform test calculation standard of the camera chip in the related technology, and realize the transverse comparison of the chip performance.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort.
FIG. 1 illustrates an exemplary application system architecture diagram of a method of testing a camera chip in an embodiment of the disclosure;
FIG. 2 is a flowchart of a method for testing a camera chip in an embodiment of the disclosure;
FIG. 3 is a flowchart showing a specific example of a method for testing a camera chip according to an embodiment of the disclosure;
FIG. 4 is a flowchart showing still another embodiment of a method for testing a camera chip according to an embodiment of the disclosure;
FIG. 5 is a flowchart showing another embodiment of a method for testing a camera chip in an embodiment of the disclosure;
FIG. 6 is a flow chart illustrating a method of unified message interface protocol transmission in an embodiment of the present disclosure;
FIG. 7 is a flowchart of a method for tamper-proof testing of a camera chip in an embodiment of the disclosure;
FIG. 8 illustrates an exemplary device architecture diagram of a method of testing a camera chip in an embodiment of the disclosure;
FIG. 9 is a schematic diagram of a test apparatus for a camera chip in an embodiment of the disclosure;
fig. 10 shows a block diagram of a computer device in an embodiment of the disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
For ease of understanding, before describing embodiments of the present disclosure, several terms referred to in the embodiments of the present disclosure are first explained as follows:
IP: internet Protocol, internetworking protocol;
linux: linux Is No UniX, operating system kernel;
PC: personal Computer, personal computer;
AI: artificial Intelligence, artificial intelligence.
The following detailed description of embodiments of the present disclosure refers to the accompanying drawings.
Fig. 1 shows a schematic diagram of an exemplary application system architecture to which a method for testing a camera chip in an embodiment of the present disclosure may be applied. As shown in fig. 1, the system architecture may include an application interaction layer 101, a logical test layer 102, a test physical layer 103, and a network 104.
The network 104 is a medium used to provide a communication link (using a unified messaging protocol) between the logical test layer 102 and the test physical layer 103, and may be a wired network or a wireless network.
The application interaction layer 101 includes an intelligent discovery and connection module 1011, an algorithm model library 1012, a test progress monitoring module 1013, and a visual reporting module 1014.
The logic test layer 102 includes a test logic module 1021 and a data parsing module 1022, the test logic module 1021 including device patrol logic 10211, a test data set 10212, and test logic 10213.
The test physical layer 103 includes a camera 1031 of vendor a, a camera 1032 of vendor B, a camera 1033 of vendor C, and a camera 1034 of vendor D. All cameras uniformly exchange message formats, and perform network data transmission by using Socket technology, and preset algorithm test models.
Optionally, the unified message interaction protocol: is compatible with most of the mainstream camera chips in China;
intelligent discovery and connection module: realizing batch terminal searching based on IP and IP sections;
device detection logic: detecting and connecting a terminal;
and a test logic module: uniformly adopting a general classical algorithm model (such as a YOLO target detection algorithm model, a MobileNet computer vision model and a ResNet residual neural network model), shielding the influence of algorithm difference on the chip performance, and directly obtaining the comprehensive performance of the software and the hardware of the chip;
test dataset: unifying the test standard to realize transverse comparison;
and a data analysis module: and (5) adapting to a Linux chip architecture to obtain chip bottom layer test data.
Those skilled in the art will appreciate that the number of cameras and networks in fig. 1 is merely illustrative, and that any number of cameras and networks may be provided as desired. The embodiments of the present disclosure are not limited in this regard.
Fig. 2 shows a flowchart of a method for testing a camera chip in an embodiment of the disclosure, and as shown in fig. 2, the method for testing a camera chip provided in the embodiment of the disclosure includes the following steps:
s202, according to the unified message interaction protocol, a connection protocol is sent to be connected with a plurality of cameras to be detected.
It should be noted that the unified message interaction protocol may be a unified interface message format, where routing and data transmission are performed between the host and the host, for example, routing and data transmission in the unified interface message format are performed between the PC end and the camera end to be detected (corresponding to the camera to be detected). The connection protocol may be Hello messages. The camera may be a camera provided with an AI algorithm, for example, an AI front-end camera, an AI intelligent monitoring camera, or an AI camera.
For example, send a hello message; when receiving the information message returned by the camera to be detected, the camera to be detected is connected with the camera to be detected.
S204, issuing chip test data to each camera to be detected, so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model.
It should be noted that the chip test data may be the same test data set. The chip may be an AI chip of the camera to be detected. The test result information may be performance test result data of the AI chip, for example, measurement index data of chip performance, such as calculation power, power consumption, area, throughput and time delay, precision, scalability and flexible applicability. The algorithm model can be an object detection AI algorithm, such as a YOLO object detection algorithm model, a MobileNet computer vision model and a ResNet residual neural network model, and concretely can be a YOLO_v5s fifth generation YOLO object detection algorithm model, a MobileNet_v2 improved computer vision model and a ResNet_v2-50 brand new depth residual neural network model.
For example, a test request message is sent, wherein the test request message comprises chip test data; and receiving a return message after the camera to be detected confirms the request.
S206, receiving test result information of the chips returned by each camera to be detected, and determining a chip test result table.
It should be noted that, the above-mentioned chip test result table includes test result information of chips returned by each camera to be detected.
For example, sending a message for acquiring a test result; obtaining a test result message, wherein the test result message comprises test result information; and determining a chip test result table according to the plurality of test result messages.
The method and the device can be compatible with chip architectures and interfaces of different manufacturers and shield the influence of algorithms on the performance of the chip by combining the unified (same) message interaction protocol with the unified (same) algorithm model, solve the problem of non-uniform test calculation standard of the camera chip in the related technology, and realize the transverse comparison of the artificial intelligence performance of the chip AI.
In an embodiment of the disclosure, the method for testing a camera chip further includes: if the chip test result of one to-be-detected camera and the preset chip performance value meet the preset conditions, judging that the to-be-detected camera is a to-be-detected cheating camera; the anti-cheating detection is carried out on the anti-cheating cameras to be detected in a preset anti-cheating mode, and anti-cheating detection results of the anti-cheating cameras to be detected are determined.
In an embodiment of the present disclosure, as shown in fig. 3, the test method for a camera chip provided in the embodiment of the present disclosure may determine whether the chip needs to perform anti-cheating detection through the following steps, so as to check whether the chip actually runs a model (uniformly set algorithm model) imported by a test system, thereby effectively avoiding chip performance data falsification:
S302, calculating a first offset value between a chip test result and a preset chip performance value;
s304, if the first offset value exceeds the preset threshold value, judging that the preset condition is met.
It should be noted that the offset value may be used to represent the degree of difference between two data of the calculated chip test result and the preset chip performance value, for example, the offset value is determined by a percentage change mode (calculation), the difference between two numbers (the chip test result and the preset chip performance value) is calculated, then the difference is divided by the average value of the original values, and finally the result is multiplied by 100 to obtain the percentage change; the offset value is determined in a standard deviation mode, the standard deviation of two groups of data (chip test results and preset chip performance values) is calculated, and then the standard deviation is divided by the average value of the average values of the two groups of data; the offset value is determined by means of a relative error, the difference between the two numbers (chip test result and preset chip performance value) is calculated, and then the difference is divided by the absolute value of the average of the two numbers. The preset threshold may be a value preset according to a calculation mode adopted by the offset value, for example, a percentage, a real number. The preset chip performance value may be set according to the performance of the current mainstream chip.
In one example, the preset anti-cheating manner includes: and modifying an algorithm model in the to-be-detected cheating camera chip and/or modifying chip test data.
In a specific embodiment, when the preset anti-cheating mode includes modifying an algorithm model in the to-be-detected cheating camera chip, anti-cheating detection is performed on the to-be-detected cheating camera through the preset anti-cheating mode, and determining an anti-cheating detection result of the to-be-detected cheating camera includes: increasing the number of layers of the algorithm model in the cheating camera chip to be detected from a first value to a second value and/or increasing the parameter information of the algorithm model from a third value to a fourth value, and testing the chip test data again to determine first inspection test result information; if the second offset value of the first test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
For example, parameters of parameter conf_thres (Confidence Threshold ) and Iou _thres (Intersect over Union Threshold, cross ratio threshold) of a modified model (corresponding to the uniformly set algorithm model) are tested by adopting a normal test data set, and the test frame rate is reduced by more than 8% (corresponding to the preset threshold) and is normal; if the frame rate is degraded abnormally (the percentage of degradation is lower than 8%), the cheating suspicion exists, and the code can be further deeply examined.
For example, the number of layers of the algorithm model is modified from three to five, and then the normal test data set is adopted for testing.
The parameters of the layer number of the algorithm model can be modified at the same time, and then the normal test data set is adopted for testing.
In a specific embodiment, when the preset anti-cheating mode includes modifying chip test data, anti-cheating detection is performed on the anti-cheating camera to be detected through the preset anti-cheating mode, and determining the anti-cheating detection result of the anti-cheating camera to be detected includes: modifying data in the chip test data according to a preset proportion; issuing modified chip test data to the cheating camera to be detected, and determining second inspection test result information; if the third offset value of the second test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
For example, changing 10% (corresponding to the preset threshold) of the classification result table of the test data set into an error value, running a test model (corresponding to the uniformly set algorithm model), checking whether the actual measurement accuracy (corresponding to the offset value) is reduced by 8% -10%, and if so, eliminating the suspected cheating; if not, the cheating suspicion exists, and the code can be further deeply examined.
In one embodiment of the disclosure, a mode of modifying the number of layers or parameter information of an algorithm model in a to-be-detected cheating camera chip to perform anti-cheating detection and a mode of modifying chip test data to perform anti-cheating detection (a superposition combination mode) can be combined to improve the accuracy of detecting cheating.
For example, the number of layers or parameter information of an algorithm model in the to-be-detected cheating camera chip is modified to perform anti-cheating detection, and when the cheating suspicion is judged (the offset value does not exceed a preset threshold value), the chip test data is modified to perform anti-cheating detection.
The method can also be that the chip test data is modified to perform anti-cheating detection, and when the suspected cheating phenomenon is judged (the offset value does not exceed the preset threshold value), the number of layers or parameter information of the algorithm model in the cheating camera chip to be detected is modified to perform anti-cheating detection.
In an embodiment of the present disclosure, as shown in fig. 4, the test method for a camera chip provided in the embodiment of the present disclosure may determine the time for issuing chip test data by the following steps, and before issuing the chip test data to each camera to be detected, determine whether to perform a chip test according to a device state (in a device idle state), so as to avoid that different chip test conditions affect a test result:
S402, polling to acquire the state of a camera to be detected;
s404, when the cameras to be detected are in an idle state, issuing chip test data to each camera to be detected;
s406, when the camera to be detected is in a non-idle state, acquiring the state of the camera to be detected again after a preset time interval.
In an embodiment of the disclosure, before the preset connection protocol is sent to connect with the plurality of cameras to be detected, the method for testing the camera chip further includes: retrieving cameras of a preset Internet Protocol (IP) network segment; and adding the retrieved camera to the equipment list, and determining the camera to be detected.
Fig. 5 is a flowchart illustrating another specific example of a method for testing a camera chip in an embodiment of the disclosure, and as shown in fig. 5, the method for testing a camera chip provided in the embodiment of the disclosure includes the following steps:
s501: the camera returns equipment information to the control platform;
s502: the control platform receives the information of the camera returned to the equipment and discovers the equipment;
s503: the control platform sends and acquires the equipment state to the camera;
s504: the camera returns to the current equipment state;
s505: the control platform judges whether the current equipment state is idle, if yes, the control platform jumps to S507, if not, the control platform jumps to S506;
S506: waiting ten seconds, jumping to S503;
s507: the control platform obtains the last inferred result;
s508: the control platform issues an inference instruction to the camera;
s509: the camera starts AI deduction, and sets the equipment state as busy;
s510: the camera deduces that the equipment state is idle;
s511: the camera caches the result, covering the last inferred result, and jumps to S507.
In one embodiment of the present disclosure, whether a camera device exists in a preset IP network segment is retrieved, and if the retrieval is successful, the device is added to a device list; transmitting connection protocol connection from the equipment in the equipment list, and jumping to the chip test module after handshake is successful; issuing a test script, a test model and test data to a test equipment storage unit; selecting a tested model, clicking a start test button to issue a test task to the camera equipment; polling to obtain an idle state of the camera equipment, and obtaining a test result after the test task of the camera equipment is completed; carrying out data analysis processing on the test result and storing the result into a data table; and opening the history record page to view the comparison result of the past test data.
Fig. 6 shows a flowchart of a method for transmitting a unified message interface protocol in an embodiment of the disclosure, and as shown in fig. 6, the method for transmitting a unified message interface protocol provided in the embodiment of the disclosure includes the following steps:
S601: the device discovery, the PC end sends a Hello message to the tested device end;
s602: the tested equipment end returns the Info information message to the PC end;
s603: starting a test, and sending a test Request message (Request) to a tested equipment end by a PC end;
s604: the tested equipment returns a message (Ack) after confirming the request;
s605: inquiring the state, and sending a state inquiry message (GetStatus) by the PC end;
s606: the tested equipment returns an equipment state message (Status);
s607: the PC end sends a message (GetREDULAST) for obtaining the test result;
s608: test Result message (Result).
Fig. 7 shows a flowchart of a method for anti-cheating testing a camera chip in an embodiment of the present disclosure, and as shown in fig. 7, the method for anti-cheating testing a camera chip provided in an embodiment of the present disclosure includes the following steps:
s701: loading an anti-cheating detection algorithm model;
s702: importing a normal test data set;
s703: judging whether the test result is consistent with the predicted value, if so, jumping to S704, and if not, jumping to S707;
s704: loading a normal algorithm model;
s705: importing a cheating prevention detection data set;
s706: judging whether the test result is consistent with the predicted value, if so, jumping to S708, and if not, jumping to S707;
S707: judging cheating and checking deeply;
s708, judging that the cheating is not performed.
The anti-cheating detection algorithm model is used for modifying the layer number or algorithm parameters of the universal model, checking whether the test result is consistent with the modified estimated result, and judging whether cheating is possible or not. If the detection result is not in the estimated range, cheating is possible, and further investigation can be performed.
For example: the parameters of the models Conf_Thres and Iou _Thres are modified, and then the normal test data set is adopted for testing, so that the test frame rate can be reduced by more than 8 percent and is normal. If the frame rate is reduced abnormally, the cheating suspicion exists, and the code can be deeply examined.
The anti-cheating detection data set adopts a normal algorithm model, and the test data set is modified, so that the test result of the anti-cheating data set adopted by the chip has obvious difference with the original data set. If the detection result is not in the estimated range, cheating is possible, and further investigation can be performed.
For example: and (3) manually changing 10% of classification result tables in the test data set classification result tables into error values, then operating the test model to check whether the actual measurement accuracy is reduced by 8% -10%, and if not, performing cheating suspicion and further checking codes.
The invention provides a camera chip AI performance detection method and system with unified interface message format, unified AI algorithm test model and test data set and unified evaluation standard, which realizes the transverse comparison of the chip AI performance; the anti-cheating detection method can check whether the chip actually runs the test system lead-in model, effectively avoid the false creation of chip performance data, and can check whether the chip test process runs the designated model and the data set according to the requirement to improve the test authority.
The present disclosure defines a set of test system and chip interface message protocol suitable for camera chip, and solves the problem of difficult adaptation of various manufacturers due to different formats. The test process adopts a unified classical and representative neural network algorithm model to test the performance of the chip, unifies the test standard, shields the influence of the algorithm model on the performance evaluation of the chip, unifies the test data set and the test standard, and realizes the transverse comparison of the AI performance of the chip.
The present disclosure provides a method for anti-cheating detection of chip performance, which dynamically modifies parameters of a general algorithm model and a test data set to make a test result and a general model test result have obvious differences. And whether the chip actually runs the test system import model is checked by evaluating whether the operation result of the chip loaded with the anti-cheating detection model and the data set is matched with the pre-estimation, so that the chip performance data is effectively prevented from being counterfeited.
Fig. 8 shows an exemplary device architecture diagram to which the method of testing a camera chip in the embodiments of the present disclosure may be applied. As shown in fig. 8, the device architecture may include a network AI camera 801, and an AI camera AI test application 802.
When the device is running, the AI camera AI test application 802 issues test scripts, modes and data sets, and issues running test script instructions to the network AI cameras; and the network AI camera returns the test data to the AI camera AI test application.
Based on the same inventive concept, the embodiment of the disclosure also provides a testing device for a camera chip, as described in the following embodiment. Since the principle of solving the problem of the embodiment of the device is similar to that of the embodiment of the method, the implementation of the embodiment of the device can be referred to the implementation of the embodiment of the method, and the repetition is omitted.
Fig. 9 shows a schematic diagram of a testing apparatus for a camera chip according to an embodiment of the disclosure, as shown in fig. 9, the apparatus includes: the system comprises a transmission connection protocol module 91, a transmission chip test data module 92, a chip test set determining module 93, a cheating prevention detection module 94, a polling camera status module 95 and a retrieval IP network segment module 96.
The sending connection protocol module 91 is configured to send a connection protocol according to a unified message interaction protocol, and connect with a plurality of cameras to be detected;
the chip test data sending module 92 is configured to send chip test data to each camera to be detected, so that each camera to be detected performs a test according to the chip test data to obtain a test result of each chip, where the cameras to be detected set the same algorithm model;
the chip test set determining module 93 is configured to receive the test result information of the chips returned by each camera to be detected, and determine a chip test result table.
In an embodiment of the present disclosure, the testing device for a camera chip further includes an anti-cheating detection module 94, configured to determine that a camera to be detected is a cheating camera to be detected if a chip test result of the camera to be detected and a preset chip performance value meet preset conditions; the anti-cheating detection is carried out on the anti-cheating cameras to be detected in a preset anti-cheating mode, and anti-cheating detection results of the anti-cheating cameras to be detected are determined.
In one embodiment of the present disclosure, the anti-cheating detection module 94 described above is further configured to: calculating a first offset value between a chip test result and a preset chip performance value; and if the first offset value exceeds the preset threshold value, judging that the preset condition is met.
In one embodiment of the present disclosure, the anti-cheating manner preset in the anti-cheating detection module 94 includes modifying an algorithm model in the chip of the cheating camera to be detected, and/or modifying chip test data.
In one embodiment of the present disclosure, the anti-cheating detection module 94 described above is further configured to: increasing the number of layers of the algorithm model in the cheating camera chip to be detected from a first value to a second value and/or increasing the parameter information of the algorithm model from a third value to a fourth value, and testing the chip test data again to determine first inspection test result information; if the second offset value of the first test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
In one embodiment of the present disclosure, the anti-cheating detection module 94 described above is further configured to: modifying data in the chip test data according to a preset proportion; issuing modified chip test data to the cheating camera to be detected, and determining second inspection test result information; if the third offset value of the second test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
In one embodiment of the present disclosure, the testing device of the camera chip further includes a polling camera status module 95, configured to poll and obtain a status of a camera to be detected; when the cameras to be detected are in an idle state, issuing chip test data to each camera to be detected; when the camera to be detected is in a non-idle state, acquiring the state of the camera to be detected again after a preset time interval.
In one embodiment of the present disclosure, the testing apparatus of the above-mentioned camera chip further includes a search IP network segment module 96 for searching for a camera of a preset internet protocol IP network segment; and adding the retrieved camera to the equipment list, and determining the camera to be detected.
In one embodiment of the present disclosure, the foregoing transmission connection protocol module 91 is specifically configured to: sending a hello message; when receiving the information message returned by the camera to be detected, the camera to be detected is connected with the camera to be detected.
In one embodiment of the present disclosure, the above-mentioned transmitting chip test data module 92 is specifically configured to: transmitting a test request message, wherein the test request message comprises chip test data; and receiving a return message after the camera to be detected confirms the request.
In one embodiment of the present disclosure, the above-mentioned chipset determining module 93 is specifically configured to: sending and obtaining a test result message; obtaining a test result message, wherein the test result message comprises test result information; and determining a chip test result table according to the plurality of test result messages.
Here, the above-mentioned transmission connection protocol module 91, the transmission chip test data module 92 and the chip test set determining module 93 correspond to S202 to S206 in the method embodiment, and the above-mentioned modules are the same as examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in the above-mentioned method embodiment. It should be noted that the modules described above may be implemented as part of an apparatus in a computer system, such as a set of computer-executable instructions.
Those skilled in the art will appreciate that the various aspects of the present disclosure may be implemented as a system, method, or program product. Accordingly, various aspects of the disclosure may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.
An electronic device 1000 according to such an embodiment of the present disclosure is described below with reference to fig. 10. The electronic device 1000 shown in fig. 10 is merely an example and should not be construed as limiting the functionality and scope of use of the disclosed embodiments.
As shown in fig. 10, the electronic device 1000 is embodied in the form of a general purpose computing device. Components of electronic device 1000 may include, but are not limited to: the at least one processing unit 1010, the at least one memory unit 1020, and a bus 1030 that connects the various system components, including the memory unit 1020 and the processing unit 1010.
Wherein the storage unit stores program code that is executable by the processing unit 1010 such that the processing unit 1010 performs steps according to various exemplary embodiments of the present disclosure described in the above section of the present specification.
For example, the processing unit 1010 may perform the following steps of the method embodiment described above: according to the unified message interaction protocol, a connection protocol is sent to be connected with a plurality of cameras to be detected; issuing chip test data to each camera to be detected so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model; and receiving test result information of the chips returned by each camera to be detected, and determining a chip test result table.
The memory unit 1020 may include readable media in the form of volatile memory units such as Random Access Memory (RAM) 10201 and/or cache memory unit 10202, and may further include Read Only Memory (ROM) 10203.
The storage unit 1020 may also include a program/utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 1030 may be representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 1000 can also communicate with one or more external devices 1040 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 1000, and/or with any device (e.g., router, modem, etc.) that enables the electronic device 1000 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 1050. Also, electronic device 1000 can communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 over the bus 1030. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with the electronic device 1000, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
In particular, according to embodiments of the present disclosure, the process described above with reference to the flowcharts may be implemented as a computer program product comprising: and the computer program is executed by the processor to realize the testing method of the camera chip.
In an exemplary embodiment of the present disclosure, a computer-readable storage medium, which may be a readable signal medium or a readable storage medium, is also provided. On which a program product is stored which enables the implementation of the method described above of the present disclosure. In some possible implementations, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the disclosure as described in the "exemplary methods" section of this specification, when the program product is run on the terminal device.
More specific examples of the computer readable storage medium in the present disclosure may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In this disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Alternatively, the program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
In particular implementations, the program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Furthermore, although the steps of the methods in the present disclosure are depicted in a particular order in the drawings, this does not require or imply that the steps must be performed in that particular order or that all illustrated steps be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform, etc.
From the description of the above embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, including several instructions to cause a computing device (may be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (14)
1. The method for testing the camera chip is characterized by comprising the following steps of:
according to the unified message interaction protocol, a connection protocol is sent to be connected with a plurality of cameras to be detected;
issuing chip test data to each camera to be detected so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model;
and receiving test result information of the chips returned by each camera to be detected, and determining a chip test result table.
2. The method for testing a camera chip according to claim 1, further comprising:
if the chip test result of one camera to be detected and the preset chip performance value meet the preset conditions, judging that the one camera to be detected is a cheating camera to be detected;
and performing anti-cheating detection on the to-be-detected cheating camera in a preset anti-cheating mode, and determining an anti-cheating detection result of the to-be-detected cheating camera.
3. The method for testing a camera chip according to claim 2, further comprising:
Calculating a first offset value between the chip test result and a preset chip performance value;
and if the first offset value exceeds a preset threshold value, judging that a preset condition is met.
4. The method for testing a camera chip according to claim 2, wherein the preset anti-cheating mode includes:
and modifying an algorithm model in the to-be-detected cheating camera chip and/or modifying chip test data.
5. The method for testing a camera chip according to claim 4, wherein when the preset anti-cheating mode includes modifying an algorithm model in a to-be-detected cheating camera chip, the anti-cheating detection is performed on the to-be-detected cheating camera through the preset anti-cheating mode, and determining an anti-cheating detection result of the to-be-detected cheating camera includes:
increasing the number of layers of the algorithm model in the to-be-detected cheating camera chip from a first value to a second value, and/or increasing parameter information of the algorithm model from a third value to a fourth value, and testing the chip test data again to determine first inspection test result information;
if the second offset value of the first test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
6. The method for testing a camera chip according to claim 4, wherein when the preset anti-cheating mode includes modifying chip test data, the anti-cheating detection is performed on the to-be-detected cheating camera by the preset anti-cheating mode, and determining the anti-cheating detection result of the to-be-detected cheating camera includes:
modifying data in the chip test data according to a preset proportion;
issuing modified chip test data to the cheating camera to be detected, and determining second inspection test result information;
if the third offset value of the second test result information and the test result information of the chip exceeds a preset threshold value, judging that the cheating camera to be detected is not cheated.
7. The method of testing a camera chip of claim 1, wherein prior to said issuing chip test data to each camera to be tested, the method further comprises:
polling to acquire the state of the camera to be detected;
when the cameras to be detected are in an idle state, chip test data are issued to each camera to be detected;
and when the camera to be detected is in a non-idle state, acquiring the state of the camera to be detected again after a preset time interval.
8. The method for testing a camera chip according to claim 1, wherein before the transmitting a preset connection protocol to connect with a plurality of cameras to be tested, the method further comprises:
retrieving cameras of a preset Internet Protocol (IP) network segment;
and adding the retrieved camera to the equipment list, and determining the camera to be detected.
9. The method for testing a camera chip according to any one of claims 1 to 8, wherein the sending a connection protocol according to a unified message interaction protocol, the connecting with a plurality of cameras to be tested includes:
sending a hello message;
when receiving the information message returned by the camera to be detected, the camera to be detected is connected with the camera to be detected.
10. The method according to any one of claims 1 to 8, wherein issuing chip test data to each camera to be tested comprises:
transmitting a test request message, wherein the test request message comprises chip test data;
and receiving a return message after the camera to be detected confirms the request.
11. The method for testing a camera chip according to any one of claims 1 to 8, wherein the receiving the test result information of the chip returned by each camera to be tested, determining the chip test result table includes:
Sending and obtaining a test result message;
obtaining a test result message, wherein the test result message comprises test result information;
and determining a chip test result table according to the plurality of test result messages.
12. The utility model provides a testing arrangement of camera chip which characterized in that includes:
the sending connection protocol module is used for sending a connection protocol according to the unified message interaction protocol and connecting with a plurality of cameras to be detected;
the chip test data module is used for issuing chip test data to each camera to be detected so that each camera to be detected executes a test according to the chip test data to obtain a test result of each chip, wherein the cameras to be detected are provided with the same algorithm model;
the chip test set determining module is used for receiving the test result information of the chips returned by each camera to be detected and determining a chip test result table.
13. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of testing a camera chip of any one of claims 1-11 via execution of the executable instructions.
14. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the method of testing a camera chip according to any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311316496.9A CN117294833A (en) | 2023-10-11 | 2023-10-11 | Camera chip testing method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311316496.9A CN117294833A (en) | 2023-10-11 | 2023-10-11 | Camera chip testing method and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117294833A true CN117294833A (en) | 2023-12-26 |
Family
ID=89256955
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311316496.9A Pending CN117294833A (en) | 2023-10-11 | 2023-10-11 | Camera chip testing method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117294833A (en) |
-
2023
- 2023-10-11 CN CN202311316496.9A patent/CN117294833A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107341098B (en) | Software performance testing method, platform, equipment and storage medium | |
| CN110069358A (en) | Call chain trace analysis method, apparatus, electronic equipment and storage medium | |
| CN113037589B (en) | Pressure testing method and device of gateway equipment, testing platform and storage medium | |
| CN103746829A (en) | Cluster-based fault perception system and method thereof | |
| CN111274154A (en) | Automatic testing method, device, equipment and storage medium | |
| CN113326181A (en) | Fuzzy test method, device and storage medium for stateful network protocol | |
| CN113114680A (en) | Detection method and detection device for file uploading vulnerability | |
| CN116112271B (en) | Session data processing method, electronic equipment and storage medium | |
| CN113127356A (en) | Pressure measurement method and device, electronic equipment and storage medium | |
| WO2021047184A1 (en) | System failure analysis and processing method and apparatus, storage medium, and electronic device | |
| CN114911706A (en) | Use case recommendation method, device, equipment and storage medium | |
| CN117435506B (en) | Fuzzy test method, electronic device and computer readable storage medium | |
| CN109634838A (en) | Position method, apparatus, storage medium and the electronic equipment of application failure | |
| CN117294833A (en) | Camera chip testing method and related equipment | |
| CN114465919B (en) | Network service testing method, system, electronic equipment and storage medium | |
| CN115292178A (en) | Test data searching method, device, storage medium and terminal | |
| CN115544202A (en) | Alarm processing method, device and storage medium | |
| CN112307271A (en) | Safety monitoring method and device for remote control service of power distribution automation system | |
| CN115994098B (en) | Analysis method and system for software operation abnormality based on artificial intelligence | |
| CN116795727B (en) | Reproduction method and device of test process, electronic equipment and storage medium | |
| CN117093465B (en) | Server log collection method, device, communication equipment and storage medium | |
| CN117439899B (en) | Communication machine room inspection method and system based on big data | |
| CN116910756B (en) | Detection method for malicious PE (polyethylene) files | |
| CN116112266B (en) | Method for identifying session data, electronic equipment and storage medium | |
| CN112000579B (en) | Software interface testing method, system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |