JP5253255B2 - Operation screen recording device - Google Patents

Description

  The present invention relates to an operation screen recording apparatus capable of recording, for example, an operation screen and an operation state when a computer apparatus is operated as an image.

  Conventionally, in order to cope with troubles when an unauthorized operation or the like is performed on a computer device, operation contents on the computer device (also referred to as a computer terminal) have been stored as a log.

  However, since it is impossible to grasp what is actually displayed on the screen only by the information based on the log, a computer operation screen recording device that saves the screen displayed on the monitor of the computer device has been proposed. (For example, refer to Patent Document 1).

  This recording apparatus repeatedly obtains operation screen information displayed on the screen based on an operation on an input unit such as a keyboard, using a central processing unit, that is, a CPU, and repeatedly acquires it in the main storage unit, that is, continuously every time it is obtained. The presence or absence of temporal change (update) of the pixel data in the image data is detected, and the operation screen of the part in which the temporal change is detected and the time data related to the update are recorded in the storage unit.

JP 2003-330761 A

  According to Patent Document 1 described above, if the operation screen is continuously recorded, the amount of recorded data becomes enormous and the load on the CPU that performs recording becomes a problem. When data is updated (changed), only that portion is recorded to reduce the amount of data and the load on the CPU.

  However, when the image data is updated, only that portion is recorded. However, the image data on the operation screen, that is, the number of pixels is so large that the image data is frequently updated. Therefore, at present, the amount of data required for recording has not been sufficiently reduced.

  Therefore, an object of the present invention is to provide an operation screen recording apparatus that can reduce the amount of data recorded on the operation screen.

In order to solve the above problems, an operation screen recording apparatus according to claim 1 of the present invention includes a computer main body, input means for inputting data to the computer main body, and a display device for displaying output data from the computer main body on the screen. A recording device for recording image data of an operation screen in a computer device having:
A camera device for photographing an operation state by an operator of the computer device;
A data fluctuation detection unit that inputs image data captured by the camera device and detects the presence or absence of fluctuations in the image data;
A data selection unit that selects image data and image data of an operation screen corresponding to the image data when a variation of the image data is detected by the data variation detection unit;
A data compression unit that performs compression processing by inputting both image data selected by the data selection unit;
A recording device for recording the compressed data compressed by the data compression unit together with at least time data;
A data search unit for searching and retrieving compressed data recorded in the recording device based on at least time;
A data decompression unit that performs the decompression process by inputting the compressed data extracted by the data retrieval unit;
The image data decompressed by the data decompression unit can be output to the display device.

According to a second aspect of the present invention, there is provided an operation screen recording apparatus comprising: a computer main body; input means for inputting data to the computer main body; and a display device for displaying output data from the computer main body on the screen. A recording device for recording image data,
A camera device for photographing an operation state by an operator of the computer device;
A data fluctuation detecting unit that inputs image data of an operation state photographed by the camera device and detects the presence or absence of fluctuation of the image data;
A data compression unit that performs compression processing by inputting the image data of the operation screen and the image data of the operation state;
A data selection unit that selects both compressed image data corresponding to the detected image data when a variation in the image data according to the operation state is detected by the data variation detection unit;
A recording device for recording the compressed data selected by the data selection unit together with at least time data;
A data search unit for searching and retrieving compressed data recorded in the recording device based on at least time;
A data decompression unit that performs the decompression process by inputting the compressed data extracted by the data retrieval unit;
The image data decompressed by the data decompression unit can be output to the display device.

  According to each of the above configurations, when the computer device is operated, the operation state is imaged by the camera device together with the operation screen, and when the image data obtained by imaging the operation state fluctuates, for example, the operation of the predetermined part of the operator Is detected, the image data relating to the shooting and the image data relating to the operation screen are compressed together, or time data is added to the image data compressed together, Since the recording is performed, the amount of recording data can be reduced as compared with the case where the image data is recorded based on the temporal change of the pixel data of the photographed image data as in the prior art.

It is a block diagram which shows schematic structure of the operation screen recording device which concerns on Embodiment 1 of this invention. It is a figure explaining the detection operation | movement in the data fluctuation | variation detection part in the same operation screen recording device. It is a block diagram which shows schematic structure of the operation screen recording device which concerns on Embodiment 2 of this invention.

[Embodiment 1]
Hereinafter, an operation screen recording apparatus in the computer apparatus according to Embodiment 1 of the present invention will be described with reference to FIGS.

This operation screen recording device records an operation screen of a computer device (also called a computer terminal) and an operation state of an operator as a video.
As shown in FIG. 1, the computer apparatus 1 includes at least a computer main body 2, an input means 3 such as a keyboard for inputting data and the like into the computer main body 2, and a display device (monitor) that is a display means for displaying output data. 4). Of course, the computer main body 2 is provided with a central processing unit (CPU) and a recording device such as a memory serving as a main storage unit and a hard disk for recording data.

  The operation screen recording device 11 is inputted with the camera device 12 for photographing the operation state (operation state) by the operator M of the computer device 1 and the image data of the analog signal output from the computer main body 2 to be digitally input. Image data of analog signals captured by the first data A / D converter 13 for converting to signal image data (image data relating to the operation screen, hereinafter also referred to as operation image data) and the camera device 12 , And a second data A / D converter 14 for converting the image data into digital signal image data (image data obtained by photographing the operation state, hereinafter also referred to as photographed image data), and the first data A / D converter 14. Data for inputting the image data converted into a digital signal by the D converter 13 and detecting the presence or absence (specifically, whether or not the variation is large) in the image data. When fluctuations are detected in the image data by the fluctuation detection unit 15 and the data fluctuation detection unit 15, the fact is input and converted into a digital signal by the image data and second data A / D conversion unit 13. The data selection unit 16 that selects the captured image data and the operation image data corresponding to the changed image data by inputting the image data, and the image data selected by the data selection unit 16 are input to perform compression processing. A data compression unit (also referred to as a data compression processing unit) 17 to perform, and compressed data compressed by the data compression unit 17 (that is, both compressed data of operation image data and photographed image data) are at least time data (time A recording device such as a hard disk (may be a memory) that records with a stamp (also known as a stamp and includes date data) 8, a data search unit 19 that searches and retrieves the compressed data recorded in the recording device 18 based on at least the time, and a data decompression unit that inputs the compressed data retrieved by the data search unit 19 and performs decompression processing ( 20 which can also be referred to as a data decompression processing unit), a data D / A conversion unit 21 that can convert both of the image data decompressed by the data decompression unit 20 from a digital signal to an analog signal, and an output from the computer main body 2 And an image switching unit (also referred to as a video switching unit) 22 for selecting any one of the image data and the image data output from the data D / A conversion unit 21 and outputting the selected image data to the display device 4. Yes.

Next, the detection operation in the data fluctuation detection unit 15 will be briefly described.
As shown in FIG. 2, in this detection operation, the image region S is divided into a plurality of, for example, 16 divisions (4 × 4), and a plurality of predetermined positions (or a predetermined number) of the divided divided region portions ΔS. This is performed for a plurality of, for example, eight divided region portions ΔS. In FIG. 2, a circle (◯) indicates a selected divided region portion, and a cross mark (×) indicates a non-selected divided region portion.

And regarding the detection of the presence or absence of a fluctuation | variation, the image data in a certain time point and the next image data will be compared.
Next, a specific method for detecting the presence or absence of fluctuation will be described.

  This detection method takes a difference in luminance value between pixels in the same divided region portion in a pair of front and rear image data (so-called two front and rear image frames), and this difference value is set in advance. When the number of pixels equal to or greater than the value (threshold) is obtained, and when the obtained number of pixels is equal to or greater than a preset value (threshold), the image data is determined to be fluctuating, , It is detected that there is motion in the image.

  As described above, when it is detected that the image data has changed, the image data at the time of the change can be specified and recorded. Of course, the image data for a predetermined time from this point can be recorded. Can also be recorded.

Here, the configuration of the data fluctuation detection unit 15 will be briefly described.
The data fluctuation detection unit 15 inputs at least a storage unit that stores image data at a certain time point and subsequent image data, and both image data stored in the storage unit, and divides each region into a plurality of regions. Luminance in pixels of a division unit, a division region selection unit that selects a predetermined division region unit from among a plurality of division region units divided by the division unit, and pixels of the division region units selected by the division region selection unit A luminance value difference calculation unit for obtaining a difference value between values, a number calculation unit for obtaining a number of difference values obtained by the luminance value difference calculation unit equal to or greater than a set value, and a number obtained by the number calculation unit are set. It is comprised from the number judgment part which judges whether it is more than a value.

  This configuration has been described so that it can be understood functionally. Normally, each of these units is realized by software operating on an operation screen recording device (that is, a computer). Each set value (threshold value) can be set to an arbitrary value, for example, an optimum value.

  In the above configuration, when the operator M is operating the computer device 1, the operation result is naturally displayed on the screen of the display device 4, and the operator M operates the camera device 12. The picture is taken.

  The operation image data sent to the display device 4 is input from the computer main body 2 to the first data A / D converter 13, converted from an analog signal to a digital signal, and photographed by the camera device 12. The captured image data is also input to the second data A / D converter 14 and converted from an analog signal to a digital signal.

  Next, the photographed image data is input to the data fluctuation detection unit 15 where the movement of the operator M is registered in advance, for example, the hand movement (specifically, keyboard input operation, mouse movement, etc.). It is detected whether or not a click operation or the like has changed significantly.

  When it is detected that there is a change, the fact is output to the data selection unit 16, where the captured image data sent via the data change detection unit 15 and an operation corresponding to the image data are performed. Image data is selected and output.

Both image data output from the data selection unit 16 are compressed by a predetermined compression method, for example, the JPEG2000 compression method.
The compressed data that has been compressed is sent to the recording device 18 and time data is added and recorded here. Of course, this time data is obtained from a timer (for example, an electronic circuit built in the computer) provided in the recording apparatus itself.

  Then, when determining whether the operation by the operator M is appropriate or not, the data search unit 19 inputs time data related to the determination and searches the compressed data in the recording device 18. When the compressed data related to the time data is retrieved and extracted, the compressed data is sent to the data decompression unit 20 where the data is decompressed (so-called decompression).

  The decompressed image data is sent to the display device 4 via the image switching unit 22, and both image data or image data is displayed separately. Therefore, it can be verified whether or not the operation of the operator M is appropriate.

As described above, when the computer device 1 is operated, the operation state is photographed by the camera device 12 together with the operation screen, and whether the operation of the predetermined part of the operator has greatly changed with respect to the image data in the operation state. If it is determined whether or not it has changed significantly, that is, it has changed greatly, the image data of the operation state and the image data of the operation screen are compressed together and time data is given, Since the recording is performed in the recording device 18, the amount of recording data can be reduced as compared with the case where image data is recorded based on the temporal change of pixel data in a captured image as in the past. it can.
[Embodiment 2]
Next, an operation screen recording apparatus according to Embodiment 2 of the present invention will be described with reference to FIG.

In the first embodiment, the data selection unit is arranged before the data compression unit. However, in the second embodiment, the data selection unit is arranged after the data compression unit.
As shown in FIG. 3, the operation screen recording device 31 includes a camera device 32 that captures an operation state (operation state) by an operator M of the computer device 1, and analog signal image data output from the computer main body 2. The first data A / D converter 33 that converts the image data into digital signal image data (image data related to the operation screen, hereinafter also referred to as operation image data) and the camera device 32 A second data A / D converter 34 for inputting analog image image data and converting it into digital signal image data (image data obtained by photographing the operation state; hereinafter also referred to as photographed image data); The image data converted into a digital signal by the 2-data A / D converter 34 is input to check whether or not there is a change in the image data (specifically, whether or not the change is large). The data fluctuation detection unit 35, the operation image data from the first data A / D conversion unit 33, and the captured image data from the second data A / D conversion unit 34 are input together to perform compression processing. When a data compression unit (also referred to as a data compression processing unit) 36 and image data compressed by the data compression unit 36 are input and a change is detected in the photographed image data by the data fluctuation detection unit 35, this is indicated. A data selection unit 37 that selects both image data related to the fluctuation and inputs both the image data selected by the data selection unit 37 at least time data (also referred to as a time stamp, including date data) Recording device 38 such as a hard disk (which may be a memory) and the compressed data recorded in this recording device 38 at least at a time A data search unit 39 that retrieves and retrieves based on the data, a data decompression unit (also referred to as a data decompression processing unit) 40 that inputs compressed data retrieved by the data search unit 39 and performs decompression processing, and decompresses by the data decompression unit 40 The image data output from the computer main body 2 and the data D / A converter 41 are output from the D / A converter 41 for data that can be converted from digital signals to analog signals and output. An image switching unit (also referred to as a video switching unit) 42 for selecting any of the image data and outputting it to the display device 4 is provided.

  With such a configuration, when it is detected that the operation state by the operator has changed with respect to the image data obtained by photographing the operation state and the image data of the operation screen together, the compression related to the change is detected. By applying time data to both the image data thus recorded and recording them in the recording device, the same effect as in the case of the first embodiment described above can be obtained.

Briefly describing this configuration, image data of an operation screen in a computer device having a computer main body, input means for inputting data to the computer main body, and a display device for displaying output data from the computer main body on a screen is recorded. A recording device,
A camera device for photographing an operation state by an operator of the computer device;
A data fluctuation detecting unit that inputs image data of an operation state photographed by the camera device and detects the presence or absence of fluctuation of the image data;
A data compression unit that performs compression processing by inputting both image data selected by the data selection unit;
A data selection unit that selects both compressed image data corresponding to the detected image data when a variation in the image data according to the operation state is detected by the data variation detection unit;
A recording device for recording the compressed data selected by the data selection unit together with at least time data;
A data search unit for searching and retrieving compressed data recorded in the recording device based on at least time;
A data decompression unit that performs the decompression process by inputting the compressed data extracted by the data retrieval unit;
The image data decompressed by the data decompression unit can be output to the display device.

  In each of the above embodiments, the operation image data from the computer main body and the photographed image data from the camera device are analog signals, but of course the image data from the computer main body and / or the camera device. The image data may be a digital signal. In this case, it is not necessary to provide the data A / D converter and the data D / A converter.

DESCRIPTION OF SYMBOLS 1 Computer apparatus 2 Computer main body 3 Input means 4 Display apparatus 11 Operation screen recording apparatus 12 Camera apparatus 13 1st data A / D conversion part 14 2nd data A / D conversion part 15 Data fluctuation detection part 16 Data selection part 17 Data compression unit 18 Recording device 19 Data search unit 20 Data decompression unit 21 Data D / A conversion unit 22 Image switching unit 31 Operation screen recording device 32 Camera device 33 First data A / D conversion unit 34 Second data A / D conversion unit 35 Data fluctuation detection unit 36 Data compression unit 37 Data selection unit 38 Recording device 39 Data search unit 40 Data decompression unit 41 Data D / A conversion unit 42 Image switching unit

Claims (2)

A recording device for recording image data of an operation screen in a computer device having a computer main body, an input means for inputting data to the computer main body, and a display device for displaying output data from the computer main body on a screen,
A camera device for photographing an operation state by an operator of the computer device;
A data fluctuation detection unit that inputs image data captured by the camera device and detects the presence or absence of fluctuations in the image data;
A data selection unit that selects image data and image data of an operation screen corresponding to the image data when a variation of the image data is detected by the data variation detection unit;
A data compression unit that performs compression processing by inputting both image data selected by the data selection unit;
A recording device for recording the compressed data compressed by the data compression unit together with at least time data;
A data search unit for searching and retrieving compressed data recorded in the recording device based on at least time;
A data decompression unit that performs the decompression process by inputting the compressed data extracted by the data retrieval unit;
An operation screen recording apparatus configured to output both image data expanded by the data expansion unit to the display device. A recording device for recording image data of an operation screen in a computer device having a computer main body, an input means for inputting data to the computer main body, and a display device for displaying output data from the computer main body on a screen,
A camera device for photographing an operation state by an operator of the computer device;
A data fluctuation detecting unit that inputs image data of an operation state photographed by the camera device and detects the presence or absence of fluctuation of the image data;
A data compression unit that performs compression processing by inputting the image data of the operation screen and the image data of the operation state;
A data selection unit that selects both compressed image data corresponding to the detected image data when a variation in the image data according to the operation state is detected by the data variation detection unit;
A recording device for recording the compressed data selected by the data selection unit together with at least time data;
A data search unit for searching and retrieving compressed data recorded in the recording device based on at least time;
A data decompression unit that performs the decompression process by inputting the compressed data extracted by the data retrieval unit;
An operation screen recording apparatus configured to output both image data expanded by the data expansion unit to the display device.

Images