CN111782876A

CN111782876A - Data processing method, device and system and storage medium

Info

Publication number: CN111782876A
Application number: CN202010618042.7A
Authority: CN
Inventors: 辛安民; 王川艳; 王治光; 陆耀
Original assignee: Hangzhou Hikrobot Technology Co Ltd
Current assignee: Hangzhou Hikrobot Technology Co Ltd
Priority date: 2020-06-30
Filing date: 2020-06-30
Publication date: 2020-10-16

Abstract

The application discloses a data processing method, a data processing device, a data processing system and a storage medium, relates to the field of industrial Internet of things, defines a new data processing framework based on an industrial camera, and realizes real-time teaching or user behavior analysis of the industrial camera based on application of the industrial camera. The data processing system comprises an industrial camera, a data processing device and a terminal, wherein the industrial camera is in wired connection with the data processing device, and the data processing device is in wireless connection with the terminal. The data processing method comprises the steps that the data processing device displays a live broadcast page, records the live broadcast page to obtain a screen recording image, compresses the screen recording image to obtain a screen recording data packet, and sends the screen recording data packet to the terminal. The live broadcast page is used for presenting a configuration process of the industrial camera in a configuration interface of the target application or presenting a process for controlling the industrial camera in an operation interface of the target application. The screen recording data packet is used for realizing real-time teaching of the industrial camera based on the target application or analyzing the behavior of the user in the target application.

Description

Data processing method, device and system and storage medium

Technical Field

The present application relates to the field of industrial internet of things, and in particular, to a data processing method, apparatus, system, and storage medium.

Background

With the development of the social industry, various industrial cameras are increasingly applied to industrial detection and production. Currently, after an image is captured by an industrial camera, the image can be transmitted to a terminal through a wired connection. The terminal displays the image so that the user can complete the detection of the workpiece to be detected according to the displayed image.

Disclosure of Invention

The application provides a data processing method, a device, a system and a storage medium, a new data processing framework is defined based on an industrial camera, and real-time teaching or user behavior analysis of the industrial camera can be realized based on application of the industrial camera. And the streaming media service can be provided to the outside, and the limitation that the industrial camera cannot provide on-demand and live broadcasting to the outside is solved.

In order to achieve the purpose, the technical scheme is as follows:

in a first aspect, the present application provides a data processing method, which is applied to a data processing system, where the data processing system includes: the system comprises at least one industrial camera, a data processing device and a terminal, wherein each industrial camera is in wired connection with the data processing device, and the data processing device is in wireless connection with the terminal. The data processing method comprises the following steps: the data processing device displays the live broadcast page, records the live broadcast page to obtain a screen recording image, compresses the screen recording image to obtain a screen recording data packet, and sends the screen recording data packet to the terminal. The live broadcast page is used for presenting a configuration process of the industrial camera in a configuration interface of the target application or presenting a process for controlling the industrial camera in an operation interface of the target application. The screen recording data packet is used for realizing real-time teaching of the industrial camera based on the target application or analyzing the behavior of the user in the target application.

The data processing method is applied to a data processing system, a new data processing framework is defined based on an industrial camera, and a data processing device is added in the data processing system, so that the data processing device can be in wired communication with the industrial camera and can be in wireless communication with a terminal. Therefore, the data processing device can compress the screen recording image obtained by recording the live broadcast page, and send the compressed screen recording data packet to the terminal, so that the terminal can realize real-time teaching of the industrial camera based on the target application according to the screen recording data packet, or analyze the behavior of the user in the target application. Therefore, real-time teaching or user behavior analysis of the industrial camera can be realized based on target application of the industrial camera, real-time teaching of the industrial camera is realized through live broadcasting, and the teaching effect of achieving twice with half the effort can be achieved when the industrial camera is updated and updated quickly. By analyzing the behavior of the user in the target application based on the screen recording data packet, the abnormal behavior of the user can be detected in real time, so that the optimal user operation process is optimized. And the method and the device can provide the streaming media service externally, and solve the problem that the industrial camera cannot provide on-demand and live broadcast externally.

In a possible design, before the "compressing the screen-recorded image", the data processing method further includes: and determining a target mode, wherein the target mode is a full data mode or a video summary mode. At this time, the method for compressing the screen recording image to obtain the screen recording data packet includes: and compressing the target image corresponding to the target mode to obtain a screen recording data packet. And when the target mode is the all-data mode, the target image is a screen recording image, and when the target mode is the video abstract mode, the target image is an image obtained by extracting key frames from the screen recording image.

In another possible design, before the "compressing the target image corresponding to the target mode", the data processing method further includes: and if the target mode is a video abstract mode, performing feature extraction on the screen recording image by adopting an image processing algorithm, and extracting key frames to obtain a target image. The key frame is an internal refresh frame in the screen recording image.

In another possible design, the method for "determining the target mode" includes: the data processing device determines the packet loss rate of communication between the data processing device and the terminal, if the packet loss rate is greater than or equal to a preset threshold value, the target mode is determined to be a video summary mode, and if the packet loss rate is less than the preset threshold value, the target mode is determined to be a full data mode. Or, the data processing device receives a request message sent by the terminal, determines the target mode to be the all-data mode if the request message is used for indicating the transmission of all images, and determines the target mode to be the video summary mode if the request message is used for indicating the transmission of partial images.

In another possible design, before the "sending a screen recording data packet to the terminal", the data processing method further includes: the data processing device determines that the account of the login target application is an administrator account when determining that the pre-stored white list account comprises the account of the login target application, determines that the account of the login target application is a common account when determining that the account of the login target application is not included in the white list account, and adds information in a preset field of a format of the screen recording data packet. The information is used for indicating a target mode and logging in an account of a target application, and the white list account is used for storing an administrator account. Correspondingly, the method for sending the screen recording data packet to the terminal comprises the following steps: and the data processing device sends the screen recording data packet containing the information to the terminal so that the terminal can obtain the mode of the screen recording data packet according to the target mode indicated by the information, and the terminal displays the target mode for a user to check. And the terminal acquires the information of the data processing device under the authority according to the authority of logging in the account of the target application indicated by the information.

In another possible design, the data processing method provided by the present application further includes: the data processing device acquires running state information of the data processing device, wherein the running state information is used for indicating the running environment of the data processing device; and compressing the running state information by adopting a preset encoding mode to obtain watermark information, and adding the watermark information into the watermark of the screen recording image. At this time, the method of "sending a screen recording data packet to a terminal" includes: the data processing device sends the screen recording data packet added with the watermark information to the terminal so that the terminal can acquire the running state information of the data processing device in real time, the terminal judges whether the running environment of the data processing device is overloaded or not according to the running state information, and if the running environment is overloaded, the terminal determines that the behavior of a user in the target application is abnormal.

In a second aspect, the present application provides a data processing system. The data processing system includes: the system comprises at least one industrial camera, a data processing device and a terminal, wherein each industrial camera is in wired connection with the data processing device, and the data processing device is in wireless connection with the terminal. Data processing apparatus for carrying out a data processing method as claimed in the first aspect and any one of its possible designs. And the terminal is used for receiving the screen recording data packet sent by the data processing device and performing real-time teaching of the industrial camera based on the target application by using the screen recording data packet or analyzing the behavior of the user in the target application based on the screen recording data packet.

In a possible design, the method for analyzing the behavior of the user in the target application based on the screen recording data packet includes: the terminal obtains a first moving track of the industrial camera calibrated by the user based on the screen recording data packet, compares the first moving track with a pre-stored calibration standard track, and determines that abnormal behaviors exist when the industrial camera is calibrated by the user if the first moving track is different from the calibration standard track.

Alternatively, the first and second electrodes may be,

the terminal obtains the input operation of the user to the target application based on the screen recording data packet, judges whether the interval time between the input operation of the user to the target application and the last input operation is greater than a preset time period, and determines that the user does not operate the target application for a long time if the interval time is greater than the preset time period.

Alternatively, the first and second electrodes may be,

and the terminal acquires a second moving track of the user operation target application based on the screen recording data packet, compares the second moving track with a pre-stored operation standard track, and determines that the user operates the target application by mistake if the second moving track is different from the operation standard track.

Alternatively, the first and second electrodes may be,

the terminal obtains watermark information based on the screen recording data packet, obtains the running state information of the data processing device by adopting a watermark extraction technology, judges whether the running environment of the data processing device is overloaded or not according to the running state information, and if the running environment is overloaded, the terminal determines that the user behavior is abnormal.

In a third aspect, the present application provides a data processing apparatus. The data processing apparatus comprises means for performing the method of the first aspect or any one of the possible design approaches of the first aspect.

In a fourth aspect, the present application provides a data processing apparatus comprising a memory and a processor. The memory is coupled to the processor. The memory is for storing computer program code, the computer program code including computer instructions. When the processor executes the computer instructions, the data processing apparatus performs the data processing method as described in the first aspect and any one of its possible designs.

In a fifth aspect, the present application provides a chip system, which is applied to a data processing apparatus; the chip system includes one or more interface circuits, and one or more processors. The interface circuit and the processor are interconnected through a line; the interface circuit is configured to receive signals from the memory of the data processing apparatus and to send signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the data processing apparatus performs the data processing method as described in the first aspect and any one of its possible designs.

In a sixth aspect, the present application provides a computer readable storage medium comprising computer instructions which, when run on a data processing apparatus, cause the data processing apparatus to perform a data processing method as in the first aspect and any one of its possible designs.

In a seventh aspect, the present application provides a computer program product comprising computer instructions which, when run on data processing apparatus, cause the data processing apparatus to perform the data processing method according to the first aspect and any of its possible design approaches.

For a detailed description of the third to seventh aspects and various implementations thereof in the present application, reference may be made to the detailed description of the first aspect and various implementations thereof; moreover, for the beneficial effects of the third aspect to the seventh aspect and various implementation manners thereof, reference may be made to beneficial effect analysis in the first aspect and various implementation manners thereof, and details are not described here.

These and other aspects of the present application will be more readily apparent from the following description.

Drawings

FIG. 1 is a block diagram of a data processing system according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a computing device according to an embodiment of the present disclosure;

fig. 3 is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 4 is a second schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 5 is a third schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of another data processing apparatus according to an embodiment of the present application.

Detailed Description

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present embodiment, "a plurality" means two or more unless otherwise specified.

First, terms/nouns referred to in the embodiments of the present application will be explained.

(1) Streaming media service

The streaming media service is a technology for transmitting video and audio on the internet in real time, wherein the streaming media service directly compresses a media file without downloading the media file and then transmits the media file in a segmented transmission mode. Streaming media technology is streaming technology, the implementation of which requires a suitable transport protocol.

(2) Encoder for encoding a video signal

An encoder refers to a program or device capable of compressing digital video.

(3) Industrial camera

Industrial cameras are a key component in machine vision systems, whose essential function is to convert light signals into ordered electrical signals.

(4) Data formats supported by industrial cameras

The data formats supported by the industrial camera may include: bayer, RGB, YUV, etc.

The Bayer format is the raw picture format inside the camera, with the suffix name of raw.

RGB is one of many color spaces, and each color in this format can be represented by three variables, red, green, and blue.

YUV is a color space, and the YUV format generally includes two types, a packed (packed) format and a planar (planar) format.

(5) Industrial camera supported protocols

When the industrial Camera performs data interaction with the data processing device, the application layer may adopt protocols such as GigE Vision, USB3.0, and Camera Link, and the transmission layer may adopt a User Datagram Protocol (UDP).

GigE Vision: the camera interface standard is developed based on a gigabit Ethernet communication protocol.

USB 3.0: is a Universal Serial Bus (USB) specification, which is initiated by intel, et al.

Camera Link: the serial communication protocol is a serial communication protocol specially for the field of machine vision application, and uses low-voltage differential signaling (LVDS) for data transmission and communication.

(6) Protocol supported by data processing apparatus

When the data processing apparatus performs data interaction with the terminal, the application layer may use a Real Time Streaming Protocol (RTSP), a Real Time Messaging Protocol (RTMP), a hypertext transfer Protocol (HTTP), an hls (HTTP live Streaming), or the like, and the Transport layer may use a UDP or Transmission Control Protocol (TCP).

The RTSP has low time delay and can be applied to security scenes.

The data processing method provided by the embodiment of the application is applied to a data processing system, and the data processing system comprises: the system comprises an industrial camera, a data processing device connected with the industrial camera in a wired mode, and a terminal connected with the data processing device in a wireless mode. In the related art, a data processing system includes only an industrial camera and a terminal to which the industrial camera is wired, and an image of the industrial camera is transmitted from the industrial camera to the terminal.

The embodiment of the application defines a new data processing framework based on the industrial camera, and can realize real-time teaching or user behavior analysis of the industrial camera based on the application of the industrial camera. And the streaming media service can be provided to the outside, and the limitation that the industrial camera cannot provide on-demand and live broadcasting to the outside is solved.

The data processing method provided by the embodiment of the application can be applied to a data processing system. Fig. 1 shows a configuration of the data processing system. As shown in fig. 1, the data processing system may include: a plurality of industrial cameras 11, electronic devices 12, and terminals 13. Wherein the electronic device 12 is connected to each industrial camera 11 by wire, and the electronic device 12 is connected to the terminal 13 wirelessly.

The industrial camera 11 may include an acquisition module and a communication module. The acquisition module is used for acquiring the shot image and transmitting the shot image to the communication module. A communication module for transmitting the photographed image to the electronic device 12. The format of the shot image acquired by the acquisition module may be a format supported by the industrial camera 11, such as Bayer, RGB, and YUV.

And an electronic device 12 for receiving the photographed image transmitted by the communication module of each industrial camera 11, compressing the photographed image, and transmitting the compressed photographed data packet to the terminal 13. The electronic device 12 is further configured to display a live broadcast page, record the live broadcast page to obtain a screen recording image, compress the screen recording image to obtain a screen recording data packet, and send the screen recording data packet to the terminal 13.

In some embodiments, the electronic device 12 may be a laptop computer, a desktop computer, a tablet computer, or the like. A Machine Vision Service Platform (MVSP) is installed on the electronic device 12, and the machine vision service platform can process and transmit a shot image or a screen recording image.

Also, the electronic device 12 may include an encoder for compressing the image to satisfy the storage or transmission of the image. Wherein the encoder may be an ffmpeg based encoder, an xvid based encoder, a VPX encoder, an AV1 encoder, etc.

And the terminal 13 is configured to receive the shot data packet sent by the electronic device 12, decode the shot data packet to obtain a shot image, and display the shot image, so that a user can complete detection of the workpiece to be detected according to the displayed shot image. The terminal 13 is further configured to receive a screen recording data packet sent by the electronic device 12, and perform real-time teaching of the industrial camera based on the target application by using the screen recording data packet, or analyze a behavior of the user in the target application based on the screen recording data packet.

In some embodiments, the terminal 13 may include a cell phone, a tablet, a laptop, a desktop, a laptop, or the like.

The basic hardware structures of the industrial camera 11, the electronic device 12 and the terminal 13 are similar and all include elements included in the computing apparatus shown in fig. 2. The hardware configuration of the industrial camera 11, the electronic device 12, and the terminal 13 will be described below by taking the computing apparatus shown in fig. 2 as an example.

As shown in fig. 2, the computing device may include a processor 21, a memory 22, a communication interface 23, and a bus 24. The processor 21, the memory 22 and the communication interface 23 may be connected by a bus 24.

The processor 21 is a control center of the computing device, and may be a single processor or a collective term for a plurality of processing elements. For example, the processor 21 may be a general-purpose CPU, or may be another general-purpose processor. Wherein a general purpose processor may be a microprocessor or any conventional processor or the like.

For one embodiment, processor 21 may include one or more CPUs, such as CPU 0 and CPU 1 shown in FIG. 2.

The memory 22 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

In a possible implementation, the memory 22 may exist separately from the processor 21, and the memory 22 may be connected to the processor 21 via a bus 24 for storing instructions or program codes. The processor 21 can implement the data processing method provided by the following embodiments of the present application when calling and executing the instructions or program codes stored in the memory 22.

In the embodiment of the present application, the software programs stored in the memory 22 are different for the industrial camera 11, the electronic device 12, and the terminal 13, so the functions implemented by the industrial camera 11, the electronic device 12, and the terminal 13 are different. The functions performed by the devices will be described in connection with the following flow charts.

In another possible implementation, the memory 22 may also be integrated with the processor 21.

The communication interface 23 is used for connecting the computing apparatus and other devices through a communication network, where the communication network may be an ethernet, a Radio Access Network (RAN), a Wireless Local Area Network (WLAN), or the like. The communication interface 23 may include a receiving unit for receiving data, and a transmitting unit for transmitting data.

The bus 24 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (enhanced Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 2, but it is not intended that there be only one bus or one type of bus.

It should be noted that the configuration shown in fig. 2 does not constitute a limitation of the computing device, which may include more or less components than those shown, or some components in combination, or a different arrangement of components than those shown in fig. 2, in addition to the components shown in fig. 2.

The execution main body of the data processing method provided by the embodiment of the application is a data processing device. The data processing device may be the electronic device 12, a CPU in the electronic device 12, a control module for processing and transmitting data in the electronic device 12, or a client for processing and transmitting data in the electronic device 12. The embodiment of the present application takes an example where the electronic device 12 executes a data processing method, and describes the data processing method provided in the present application.

The data processing method provided by the embodiment of the present application is described below with reference to the drawings.

As shown in fig. 3, the data processing method provided by the embodiment of the present application includes the following steps 301 to 304.

301. The electronic device displays a live page.

The live broadcast page is used for presenting a configuration process of the industrial camera in a configuration interface of the target application or presenting a process for controlling the industrial camera in an operation interface of the target application.

It is to be understood that, in the embodiments of the present application, the target application refers to an application corresponding to an industrial camera installed on an electronic device. The configuration interface of the target application refers to an interface for a user to perform parameter configuration on the industrial camera, and the operation interface of the target application refers to an interface for the user to control the industrial camera.

302. And the electronic equipment records the live broadcast page to obtain a recorded screen image.

The electronic equipment can record the live broadcast page displayed by the electronic equipment in real time, so that a recorded screen image is obtained. At this time, the action track of the user in the target application is included in the screen recording image.

303. And the electronic equipment compresses the screen recording image to obtain a screen recording data packet.

After the electronic device obtains the screen recording image, the electronic device may compress the screen recording image by using an encoder to obtain a compressed screen recording data packet.

Optionally, in this embodiment of the present application, the data format of the screen recording image may be an encapsulation format using a compression algorithm, such as MP4, AVI, RTP, and the like. When the electronic device compresses the screen recording image, the problem that the data format of the screen recording image is different from the data format supported by the encoder may exist, so that the electronic device may convert the format of the screen recording image, that is, convert the format of the screen recording image into the format supported by the encoder, and then compress the screen recording image in the format supported by the encoder. Therefore, the successful compression of the screen recording image can be ensured.

Optionally, in this embodiment of the application, before compressing the screen recording image, the electronic device may adjust the resolution of the screen recording image to a preset resolution, and/or adjust the image quality parameter of the screen recording image to a preset image quality parameter, and then compress the adjusted screen recording image, so as to obtain the screen recording data packet. The image quality parameters may include brightness, sharpness, denoising, and the like.

It can be understood that, in the embodiment of the present application, the preset resolution and the preset image quality parameter may be pre-stored in the electronic device, or may be generated by the terminal according to a requirement of a user and sent to the electronic device. Therefore, the electronic equipment adjusts the screen recording image, and after the adjusted screen recording image is sent to the terminal, the terminal can display the adjusted screen recording image, so that the requirements of users on resolution or image quality parameters and specific scene requirements are met.

304. And the electronic equipment sends a screen recording data packet to the terminal.

After obtaining the screen recording data packet, the electronic device may send the screen recording data packet to the terminal. The screen recording data packet is used for realizing real-time teaching of the industrial camera based on the target application, or the screen recording data packet is used for analyzing the behavior of the user in the target application.

Optionally, in this embodiment of the present application, the data processing method may further include: the electronic device may obtain running state information of itself, where the running state information is used to indicate a running environment of the electronic device, and the running state information may specifically include an occupancy rate of a CPU or a memory. And then, the electronic equipment can compress the running state information by adopting a preset coding mode to obtain watermark information, and the watermark information is added into the watermark of the screen recording image. In this scenario, the sending, by the electronic device, the screen recording data packet to the terminal may specifically include: and the electronic equipment sends the screen recording data packet added with the watermark information to the terminal. Therefore, the terminal can conveniently acquire the running state information of the electronic equipment in real time, the terminal judges whether the running environment of the electronic equipment is overloaded or not according to the running state information, and if the running environment is overloaded, the terminal determines that the behavior of the user in the target application is abnormal.

It can be understood that the preset encoding manner is to encrypt the operation state information to ensure the security of the information.

Optionally, in this embodiment of the application, with reference to fig. 3, as shown in fig. 4, before performing step 303, the data processing method further includes the following step 305.

305. The electronic device determines a target mode.

The target mode may be a full data mode or a video summary mode.

Optionally, in this embodiment of the application, the electronic device may determine the target mode in the following two ways.

In the mode 1, the target mode is determined according to the packet loss rate.

In a specific implementation, the electronic device may send a preset number of detection packets to the terminal within a preset time period, and if the terminal successfully receives the detection packets, the terminal may send a feedback message indicating that the detection packets are successfully received to the electronic device. In this way, the electronic device may calculate a packet loss rate of the communication between the electronic device and the terminal by using a formula, where the packet loss rate is (N-M)/N. Wherein, N is the number of the detection packets sent by the electronic device within the preset time period, and M is the number of the feedback messages received by the electronic device within the specified time period. If the packet loss rate is greater than or equal to the preset threshold value, the current network quality is poor, and at the moment, the electronic equipment determines that the target mode is the video abstract mode. If the packet loss rate is smaller than the preset threshold value, the current network quality is better, and at the moment, the electronic equipment determines that the target mode is the all-data mode.

Mode 2, the target mode is determined according to a request message sent by the terminal.

In a specific implementation, the electronic device may receive a request message sent by the terminal. If the request message is used to indicate that the entire image is transmitted, the electronic device determines that the target mode is the entire data mode. If the request message is used to indicate transmission of the partial image, the electronic device determines that the target mode is a video summary mode.

The terminal can actively send the detection packets to the electronic device and receive the feedback messages sent by the electronic device and used for indicating successful reception of the detection packets, so that the packet loss rate is determined according to the number of the detection packets and the number of the feedback messages. And the terminal outputs prompt information according to the packet loss rate, wherein the prompt information is used for prompting the condition of the current network quality of the user and prompting the user to select a mode of transmitting the image corresponding to the current network quality. After the user confirms the operation at the terminal, the terminal can send a request message to the electronic device.

It can be understood that, compared with the mode 1 and the mode 2, in the mode 2, the user can autonomously select the mode of the video to be watched, the user experience is better, and different requirements of the user can be met.

As shown in fig. 4, after the step 305 is executed, the step 303 may specifically include the following step 303 a.

303a, the electronic device compresses the target image corresponding to the target mode to obtain a screen recording data packet.

After the electronic device determines the target mode, if the target mode is the full data mode, the electronic device may determine that the target image is a screen recording image. If the target mode is a video abstract mode, the electronic equipment can extract the characteristics of the screen recording image by adopting an image processing algorithm, so that a key frame is extracted from the screen recording image to obtain the target image. And the key frame is an internal refresh frame in the screen recording image. And then the electronic equipment can compress the target image to obtain a compressed screen recording data packet.

It is to be understood that the above-described intra-refresh frame refers to an I-frame in a video compression IPB frame. I-frames refer to frames in which a complete image is retained. When decoding, only I frame is used, and the complete image can be reconstructed without referring to other frames.

Optionally, in this embodiment of the application, the account on which the target application is logged in on the electronic device may be a common account or an administrator account, and the permissions of different accounts are different. Therefore, before performing step 304, the electronic device may determine an account for logging in the target application. Specifically, the electronic device may determine that the account logged in the target application is an administrator account when determining that the pre-stored white list account includes an account logged in the target application. The electronic device may determine that the account logged in to the target application is a common account when determining that the account logged in to the target application is not included in the white list account. The white list account is used for storing an administrator account. Then, the electronic device may add information in a preset field of the format of the screen recording data packet, where the information is used for indicating the target mode and logging in the account of the target application.

In this scenario, the sending, by the electronic device, the screen recording data packet to the terminal in step 304 may specifically include: the electronic equipment sends the screen recording data packet containing the information to the terminal, so that the terminal can obtain the mode of the screen recording data packet according to the target mode indicated by the information, and the terminal displays the target mode for a user to check. And the terminal acquires the information of the electronic equipment under the authority according to the authority indicated by the information for logging in the account of the target application. For example, if the account of the login target application is an administrator account, the terminal has the right to acquire the watermark information of the screen recording data packet, so as to acquire the running state information of the electronic device.

Fig. 5 is a flowchart of a data processing method provided in an embodiment of the present application, and when the data processing method is applied to the data processing system shown in fig. 1, as shown in fig. 5, the data processing method includes steps 401 to 408.

401. The electronic device displays a live page.

402. And the electronic equipment records the live broadcast page to obtain a recorded screen image.

The detailed descriptions of steps 401 to 402 may refer to the related descriptions of steps 301 to 302 in another embodiment, which are not repeated herein.

403. The electronic device determines a target mode.

404. And the electronic equipment compresses the target image corresponding to the target mode to obtain a screen recording data packet.

The detailed descriptions of step 403 to step 404 may refer to the related descriptions of step 305 to step 305a in another embodiment, which are not repeated herein.

405. The electronic equipment determines an account for logging in the target application, and adds information in a preset field of a format of the screen recording data packet, wherein the information is used for indicating a target mode and the account for logging in the target application.

The specific process of step 405 may refer to another embodiment to determine an account for logging in the target application, and add a relevant description of information in a preset field of the format of the screen recording data packet, which is not described herein again.

406. And the electronic equipment sends a screen recording data packet containing information to the terminal.

407. And the terminal receives a screen recording data packet which is sent by the electronic equipment and contains information.

408. And the terminal uses the screen recording data packet to perform real-time teaching of the industrial camera based on the target application, or analyzes the behavior of the user in the target application based on the screen recording data packet.

Optionally, in this embodiment of the application, the analyzing, by the terminal, the behavior of the user in the target application based on the screen recording data packet may specifically include:

the terminal obtains a first moving track of the industrial camera calibrated by the user based on the screen recording data packet, compares the first moving track with a pre-stored calibration standard track, and if the first moving track is different from the calibration standard track, the terminal determines that abnormal behaviors exist when the industrial camera is calibrated by the user. It is to be understood that the user calibrating the industrial camera is part of the user configuring the industrial camera in the target application, and that the process of the user configuring the industrial camera includes other configuration processes besides calibrating the industrial camera.

Alternatively, the first and second electrodes may be,

the terminal acquires the input operation of the user on the target application based on the screen recording data packet, judges whether the interval time between the input operation of the user on the target application and the last input operation is greater than a preset time period, and if the interval time is greater than the preset time period, the terminal determines that the user does not operate the target application for a long time.

Alternatively, the first and second electrodes may be,

the terminal obtains a second moving track of the user operation target application based on the screen recording data packet, compares the second moving track with a pre-stored operation standard track, and if the second moving track is different from the operation standard track, the terminal determines that the user operates the target application by mistake.

Alternatively, the first and second electrodes may be,

the terminal obtains watermark information based on the screen recording data packet, obtains the running state information of the electronic equipment by adopting a watermark extraction technology, judges whether the running environment of the electronic equipment is overloaded or not according to the running state information, and if the running environment is overloaded, the terminal determines that the user behavior is abnormal.

In order to facilitate understanding of those skilled in the art, the embodiments of the present application describe a data processing method provided by the embodiments of the present application according to different application scenarios.

Scene one, the live teaching scene of novel industry camera: and real-time teaching is carried out on the using process of the novel industrial camera.

Specifically, the administrator starts a screen recording function of the electronic device, starts a target application installed on the electronic device, performs teaching and training on a management attribute of the industrial camera in a configuration interface of the target application, performs teaching and training on a control process of the industrial camera in an operation interface of the target application, and closes the screen recording function after the operation is finished. The operation behavior of the administrator can be captured by the electronic equipment in real time, the electronic equipment obtains the screen recording image, the target image is determined according to the determined target mode, and then the target image is coded to obtain a screen recording data packet. And finally, the electronic equipment sends the screen recording data packet to the terminal.

And a second scenario is a user behavior analysis scenario of the industrial camera operation. The electronic equipment can send a screen recording data packet to the terminal. The terminal can detect user behavior of industrial camera operations to determine which steps are erroneous or prone to error. Therefore, research and development personnel can be helped to acquire the first-hand user information, and therefore the optimal user operation process is optimized.

Specifically, in a standard detection process of the mobile phone shell, a common user starts a screen recording function of the electronic device, starts a target application installed on the electronic device, and then calibrates the industrial camera in the target application and performs other configurations except for calibration. And then, entering an abnormal detection page of the mobile phone shell, connecting the industrial camera, controlling the industrial camera to work in an operation interface of the target application, closing a screen recording function after user operation is finished, obtaining a screen recording image by the electronic equipment, determining the target image according to the determined target mode, and coding the target image to obtain a screen recording data packet. And finally, the electronic equipment sends the screen recording data packet to the terminal.

After receiving the screen recording data packet, the terminal can analyze the behavior of the user in the target application based on the screen recording data packet. For example: the terminal can analyze whether abnormal behaviors exist when the user calibrates the industrial camera, whether the target application is not operated for a long time, whether the target application is used by mistake, and whether the running environment of the electronic equipment is overloaded due to the abnormal behaviors of the user.

Optionally, in this embodiment of the application, the electronic device, in addition to implementing the screen recording function, provides a data source for the streaming media service: besides recording the screen image, the electronic equipment can also receive a shot image sent by an industrial camera, and provides another data source for the streaming media service: an image is captured. The data processing method may further include:

the electronic device receives a captured image transmitted by the industrial camera.

The electronic equipment compresses the shot image to obtain a shot data packet.

Optionally, in this embodiment of the application, the electronic device compresses the captured image, and a specific process of obtaining the captured data packet is as follows: after receiving the shot image sent by the industrial camera, the electronic device can convert the shot image from the first format to the second format, and compress the shot image in the second format to obtain a shot data packet. The first format is a data format supported by the industrial camera, and the second format is a data format supported by the electronic device, that is, the second format is a format supported by an encoder included in the electronic device. Therefore, the problems that encoding fails and user experience is poor due to the fact that the data format supported by the industrial camera is different from the data format supported by the encoder can be solved. For example, the data format supported by the industrial camera is Bayer, RGB, YUV, etc., and the data format supported by the encoder is only YUV.

Optionally, in this embodiment of the application, before compressing the captured image, the electronic device may adjust the resolution of the captured image to a preset resolution, and/or adjust the image quality parameter of the captured image to a preset image quality parameter, and then compress the adjusted captured image, so as to obtain the captured data packet, so as to meet the requirement of the user on the resolution or the image quality parameter and meet the requirement of the specific scene.

The electronic equipment sends a shooting data packet to the terminal.

In the embodiment of the present application, the execution subject for adjusting the resolution and the image quality parameters of the captured image may be an electronic device or a terminal.

In a first implementation, the execution subject of the adjustment image is an electronic device. The electronic device can adjust the resolution, image quality parameters and the like of the shot image, compress the adjusted shot image and send a compressed shot data packet to the terminal. And after receiving the shooting data packet, the terminal decompresses the shooting data packet to obtain an adjusted shooting image, and displays the adjusted shooting image.

In a second implementation, the execution subject of the adjustment image is the terminal. The electronic device may directly compress the shot image received from the industrial camera and transmit the compressed shot data packet to the terminal. After receiving the shooting data packet, the terminal decompresses the shooting data packet to obtain a shooting image, adjusts the resolution, image quality parameters and the like of the shooting image, and displays the adjusted shooting image.

It can be understood that, compared with the above two implementation manners, firstly, the first manner increases the processing pressure of the electronic device on the captured image, and the second manner increases the processing pressure of the terminal on the captured image. Next, in a case where the adjusted captured image is smaller than the data amount of the captured image before adjustment, for example, in a case where the resolution of the captured image is adjusted to be small, the first mode can reduce the data amount of the captured image to be transmitted, thereby saving the transmission bandwidth. In the embodiment of the present application, what kind of device is specifically used for adjusting the execution subject of the captured image may be set in advance according to an actual situation, and the embodiment of the present application is not limited herein.

Optionally, in this embodiment of the application, when the shot image includes the workpiece to be detected, the data processing method can implement remote detection of the workpiece to be detected. In the scene, the electronic device can perform image processing on the shot image by adopting a preset algorithm, determine whether the workpiece to be detected included in the shot image is abnormal, and send a detection instruction to the terminal according to whether the workpiece to be detected is abnormal, so that the terminal executes different operations according to the detection instruction.

For example, in a handset shell standard detection procedure, an electronic device receives a handset shell image taken by an industrial camera. And the electronic equipment performs image processing on the mobile phone shell image, determines whether the mobile phone shell in the mobile phone shell image has defects, generates a failure instruction if the mobile phone shell in the mobile phone shell image has defects, and generates a success instruction if the mobile phone shell in the mobile phone shell image does not have defects. The electronic device may send different instructions to the associated hardware apparatus. For example, the electronic device may send a failure instruction to a hardware device, such as a robot, to instruct the robot to grab a defective handset housing to the review area in accordance with the failure instruction.

According to the data processing method, the data processing device can compress the shot images received from the industrial camera and send the compressed data packets to the terminal. The remote wireless transmission of the shot images is realized, so that the streaming media service can be provided to the outside.

The scheme provided by the embodiment of the application is mainly introduced from the perspective of a method. To implement the above functions, it includes hardware structures and/or software modules for performing the respective functions. Those of skill in the art will readily appreciate that the present application is capable of hardware or a combination of hardware and computer software implementing the various illustrative algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The embodiment of the application also provides a data processing device. The data processing apparatus may be the electronic device 12, or may be a CPU in the electronic device 12, or may be a control module in the electronic device 12, or may be a client in the upper electronic device 12.

Fig. 6 is a schematic structural diagram of a data processing apparatus 50 according to an embodiment of the present disclosure. The data processing apparatus 50 is used to execute the data processing method shown in fig. 3 or fig. 4. The data processing device 50 may include a display unit 51, a recording unit 52, an encoding unit 53, and a transmitting unit 54.

The display unit 51 is configured to display a live page. For example, in connection with fig. 3, the display unit 51 may be used to perform step 301. And the recording unit 52 is used for recording the live broadcast page to obtain a recorded screen image. For example, in conjunction with fig. 3, recording unit 52 may be configured to perform step 302. And the encoding unit 53 is configured to compress the screen recording image to obtain a screen recording data packet. For example, in connection with fig. 3, the encoding unit 53 may be configured to perform step 303. And a sending unit 54, configured to send the screen recording data packet obtained by the encoding unit 53 to the terminal. For example, in connection with fig. 3, the sending unit 54 may be configured to perform step 304.

Optionally, as shown in fig. 7, the data processing apparatus may further include: a determination unit 55.

A determination unit 55 for determining the target mode. For example, in connection with fig. 4, the determination unit 55 may be configured to perform step 305. And the encoding unit 53 is further configured to compress the target image corresponding to the target mode to obtain a screen recording data packet. For example, in connection with fig. 4, the encoding unit 53 may be configured to perform step 303 a.

Optionally, the determining unit 55 is specifically configured to, if the target mode is a video summary mode, perform feature extraction on the screen recording image by using an image processing algorithm, and extract a key frame to obtain the target image.

Optionally, the determining unit 55 is specifically configured to determine a packet loss rate of communication between the data processing apparatus and the terminal, determine that the target mode is the video summary mode if the packet loss rate is greater than or equal to a preset threshold, and determine that the target mode is all data modes if the packet loss rate is less than the preset threshold. Or receiving a request message sent by the terminal, if the request message is used for indicating to transmit all images, determining that the target mode is an all-data mode, and if the request message is used for indicating to transmit partial images, determining that the target mode is a video summary mode.

Optionally, as shown in fig. 7, the data processing apparatus may further include: an adding unit 56.

The determining unit 55 is further configured to determine that the account of the login target application is an administrator account when it is determined that the pre-stored white list account includes the account of the login target application, and determine that the account of the login target application is a common account when it is determined that the account of the login target application is not included in the white list account. And the adding unit 56 is used for adding information in a preset field of the format of the screen recording data packet. The sending unit 54 is specifically configured to send a screen recording data packet including information to the terminal.

Optionally, as shown in fig. 7, the data processing apparatus may further include: an acquisition unit 57.

Wherein the obtaining unit 57 is configured to obtain the operation state information of the data processing apparatus. The encoding unit 53 is further configured to compress the operation state information acquired by the acquiring unit 57 by using a preset encoding method, so as to obtain watermark information. And the adding unit 56 is also used for adding the watermark information to the watermark of the screen recording image. And a sending unit 54, specifically configured to send the screen recording data packet to which the watermark information is added to the terminal.

Of course, the data processing apparatus 50 provided in the embodiment of the present application includes, but is not limited to, the above modules.

In actual implementation, the display unit 51, the recording unit 52, the encoding unit 53, the determining unit 55, the adding unit 56, and the obtaining unit 57 may be implemented by the processor 21 shown in fig. 2 calling program codes in the memory 22. For a specific implementation process, reference may be made to the description of the data processing method portion shown in fig. 3 or fig. 4, which is not described herein again.

Another embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions that, when executed on a data processing apparatus, cause the data processing apparatus to perform the steps performed by the data processing apparatus in the method flow shown in the above method embodiment.

Another embodiment of the present application further provides a chip system, and the chip system is applied to a data processing apparatus. The system-on-chip includes one or more interface circuits, and one or more processors. The interface circuit and the processor are interconnected by a line. The interface circuit is configured to receive signals from a memory of the data processing apparatus and to transmit the signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the data processing apparatus performs the steps performed by the data processing apparatus in the method flow illustrated in the above-described method embodiments.

In another embodiment of the present application, there is also provided a computer program product, which includes instructions that, when executed on data processing apparatus, cause the data processing apparatus to perform the steps performed by the data processing apparatus in the method flow shown in the above-mentioned method embodiment.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The processes or functions according to the embodiments of the present application are generated in whole or in part when the computer-executable instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). Computer-readable storage media can be any available media that can be accessed by a computer or can comprise one or more data storage devices, such as servers, data centers, and the like, that can be integrated with the media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

The foregoing is only illustrative of the present application. Those skilled in the art can conceive of changes or substitutions based on the specific embodiments provided in the present application, and all such changes or substitutions are intended to be included within the scope of the present application.

Claims

1. A data processing method is applied to a data processing system, and the data processing system comprises: at least one industry camera, data processing apparatus and terminal, wherein every industry camera all with data processing apparatus wired connection, data processing apparatus with the terminal wireless connection, its characterized in that includes:

displaying a live broadcast page, wherein the live broadcast page is used for presenting a configuration process of an industrial camera in a configuration interface of a target application or presenting a process for controlling the industrial camera in an operation interface of the target application;

recording the live broadcast page to obtain a screen recording image;

compressing the screen recording image to obtain a screen recording data packet;

and sending the screen recording data packet to the terminal, wherein the screen recording data packet is used for realizing real-time teaching of the industrial camera based on the target application, or the screen recording data packet is used for analyzing the behavior of the user in the target application.

2. The data processing method of claim 1, wherein prior to the compressing the screen-recorded image, the data processing method further comprises:

determining a target mode, wherein the target mode is a full data mode or a video summary mode;

compressing the screen recording image to obtain a screen recording data packet, including:

compressing a target image corresponding to the target mode to obtain the screen recording data packet; and when the target mode is the all-data mode, the target image is the screen recording image, and when the target mode is the video abstract mode, the target image is an image obtained by extracting a key frame from the screen recording image.

3. The data processing method according to claim 2, further comprising, before the compressing the target image corresponding to the target mode:

if the target mode is the video abstract mode, extracting the characteristics of the screen recording image by adopting an image processing algorithm, and extracting the key frame to obtain the target image; the key frame is an internal refresh frame in the screen recording image.

4. A data processing method according to claim 2 or 3, wherein said determining a target pattern comprises:

determining a packet loss rate of the data processing device communicating with the terminal, if the packet loss rate is greater than or equal to a preset threshold, determining that the target mode is the video summary mode, and if the packet loss rate is less than the preset threshold, determining that the target mode is the all data modes;

alternatively, the first and second electrodes may be,

and receiving a request message sent by the terminal, if the request message is used for indicating to transmit all images, determining that the target mode is the all-data mode, and if the request message is used for indicating to transmit partial images, determining that the target mode is the video summary mode.

5. The data processing method according to claim 2 or 3, wherein before the sending the screen recording data packet to the terminal, the method further comprises:

when determining that the pre-stored white list account comprises an account for logging in the target application, determining that the account for logging in the target application is an administrator account, and when determining that the white list account does not comprise the account for logging in the target application, determining that the account for logging in the target application is a common account; the white list account is used for storing an administrator account;

adding information in a preset field of the format of the screen recording data packet; the information is used for indicating the target mode and an account for logging in the target application;

the sending the screen recording data packet to the terminal includes:

and sending the screen recording data packet containing the information to the terminal so that the terminal can obtain the mode of the screen recording data packet according to the target mode indicated by the information, the terminal displays the target mode for a user to check, and the terminal obtains the information of the data processing device under the authority according to the authority indicated by the information for logging in the account of the target application.

6. A data processing method according to any one of claims 1 to 3, characterized in that the data processing method further comprises:

acquiring running state information of the data processing device; the running state information is used for indicating the running environment of the data processing device;

compressing the running state information by adopting a preset encoding mode to obtain watermark information;

adding the watermark information to the watermark of the screen recording image;

the sending the screen recording data packet to the terminal includes:

and sending the screen recording data packet added with the watermark information to the terminal so that the terminal can acquire the running state information of the data processing device in real time, judging whether the running environment of the data processing device is overloaded or not by the terminal according to the running state information, and determining that the behavior of the user in the target application is abnormal by the terminal if the running environment is overloaded.

7. A data processing system, characterized in that the data processing system comprises: at least one industrial camera, a data processing device and a terminal;

each industrial camera is in wired connection with the data processing device, and the data processing device is in wireless connection with the terminal;

the data processing apparatus for performing the data processing method of any one of claims 1-6;

and the terminal is used for receiving the screen recording data packet sent by the data processing device and performing real-time teaching on the industrial camera based on target application by using the screen recording data packet, or analyzing the behavior of a user in the target application based on the screen recording data packet.

8. The data processing system of claim 7, wherein analyzing the behavior of the user in the target application based on the screenshot data packet comprises:

acquiring a first moving track of the industrial camera calibrated by the user based on the screen recording data packet, comparing the first moving track with a pre-stored calibration standard track, and determining that abnormal behaviors exist when the industrial camera is calibrated by the user if the first moving track is different from the calibration standard track;

alternatively, the first and second electrodes may be,

acquiring the input operation of the user on the target application based on the screen recording data packet, judging whether the input operation of the user on the target application is more than a preset time period from the interval time of the last input operation, and if the interval time is more than the preset time period, determining that the user does not operate the target application for a long time;

alternatively, the first and second electrodes may be,

acquiring a second moving track of the target application operated by the user based on the screen recording data packet, comparing the second moving track with a pre-stored operation standard track, and determining that the target application is operated by the user by mistake if the second moving track is different from the operation standard track;

alternatively, the first and second electrodes may be,

acquiring watermark information based on the screen recording data packet, acquiring running state information of the data processing device by adopting a watermark extraction technology, judging whether the running environment of the data processing device is overloaded or not according to the running state information, and if the running environment is overloaded, determining that the user behavior is abnormal.

9. A data processing apparatus for use in a data processing system, the data processing system comprising: at least one industrial camera, data processing apparatus and terminal, wherein every industrial camera all with data processing apparatus wired connection, data processing apparatus with the terminal wireless connection, its characterized in that includes:

the display unit is used for displaying a live broadcast page, and the live broadcast page is used for presenting a configuration process of an industrial camera in a configuration interface of a target application or presenting a process of controlling the industrial camera in an operation interface of the target application;

the recording unit is used for recording the live broadcast page displayed by the display unit to obtain a recorded screen image;

the coding unit is used for compressing the screen recording image recorded by the recording unit to obtain a screen recording data packet;

and the transmitting unit is used for transmitting the screen recording data packet obtained by compression of the encoding unit to the terminal, wherein the screen recording data packet is used for realizing real-time teaching of the industrial camera based on the target application, or the screen recording data packet is used for analyzing the behavior of a user in the target application.

10. A data processing apparatus, characterized in that the data processing apparatus comprises a memory and a processor; the memory and the processor are coupled; the memory for storing computer program code, the computer program code comprising computer instructions; the data processing apparatus, when executing the computer instructions, performs the data processing method of any one of claims 1-6.

11. A computer-readable storage medium comprising computer instructions which, when run on data processing apparatus, cause the data processing apparatus to perform a data processing method according to any one of claims 1 to 6.