CN108229481A - Screen content analysis method, device, computing device and storage medium - Google Patents
Screen content analysis method, device, computing device and storage medium Download PDFInfo
- Publication number
- CN108229481A CN108229481A CN201711423613.6A CN201711423613A CN108229481A CN 108229481 A CN108229481 A CN 108229481A CN 201711423613 A CN201711423613 A CN 201711423613A CN 108229481 A CN108229481 A CN 108229481A
- Authority
- CN
- China
- Prior art keywords
- text
- frame
- area
- application
- word
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/126—Character encoding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Abstract
The embodiment of the invention discloses method, apparatus, computing device and the storage mediums of a kind of analysis of screen content.This method includes：Based on application image model library, active window region and its corresponding application in recognition screen image；Active window region division is one or more text areas and non-textual area by the application image model based on application；And the word in identification text area, obtain character string text.By technical scheme of the present invention, the accuracy of O＆M operation identification can be improved, promotes the efficiency of IT O＆Ms operation audit.
Description
Technical field
The present invention relates to IT system O＆M technical field more particularly to a kind of method, apparatus of screen content analysis, meters
Calculate equipment and storage medium.
Background technology
At present, in IT system O＆M operation audit, the film recording of O＆M terminal is a kind of common auditing method.But
It is that during curent audit, especially when carrying out security incident retrospect, image, audit behaviour are operated in face of the O＆M of magnanimity
Work person can not carry out retrieval audit based on the operation content in video recording, need to check video recording, inefficiency frame by frame.
Therefore, it is necessary to a kind of screen content analysis methods that can improve audit efficiency.
Invention content
An embodiment of the present invention provides a kind of screen content analysis method, device, computing device and storage mediums, can
The accuracy of O＆M operation identification is improved, promotes the efficiency of IT O＆Ms operation audit.
In a first aspect, an embodiment of the present invention provides a kind of screen content analysis method, method includes：
Based on application image model library, active window region and its corresponding application in recognition screen image；
Active window region division is one or more text areas and non-textual by the application image model based on application
Area；And
It identifies the word in text area, obtains character string text.
Be preferably based on the application image model of application by active window region division be one or more text areas and
The step of non-textual area, includes：
In the case that known to the window interface pattern of application, the window interface pattern based on application is by active window mouth region
Domain is divided into one or more text areas and non-textual area；And/or
In the case where the window interface pattern of application is unknown, the inherent feature based on text and picture is by active window
Region division is one or more text areas and non-textual area, and is updated in the application image model according to division result
The window interface pattern of record.
Preferably, identify that the step of word in text area obtains character string text includes：
To text area into every trade cutting, line of text image is obtained；
Word segmentation is carried out based on separator, obtains word or expression image；
Character cutting is carried out based on grammer and empirical model, obtains character picture；
The feature vector of calculating character image；
The corresponding character of feature based vector identification character picture or word.
Optionally, with reference to word segmentation and row cutting as a result, being combined to obtain character string text to character or word.
Optionally, based on the word in preset one or more language identification text areas.
Preferably, this method further includes：
It determines with applying corresponding business scenario；
The text identified is handled according to business scenario.
Optionally, it is one or more parts by the text classification identified according to business scenario.
Preferably, which further includes：
The non-duplicate frame in film recording is extracted, screen picture is the screen picture of non-repeating frame.
Preferably, the step of extracting the non-duplicate frame in film recording includes：
Since current non-duplicate frame, with the initial sample interval of setting, cyclic samples are carried out, are taken out twice until adjacent
The frame number of the frame of sample is less than or equal to predetermined threshold, using it is adjacent sample twice in the frame sampled of forward direction as next
Non-duplicate frame, wherein, cyclic samples include：
In response to finding the frame with current non-duplicate frame dissmilarity in sampling in forward direction, will halve in the sampling interval, and hold
The backward sampling of row；
In response to finding the frame similar to current non-duplicate frame into sampling rear, will halve in the sampling interval, and perform
Forward direction is sampled.
Preferably, this method further includes：In database table record screen picture corresponding to frame number and in screen map
The character string text identified as in.
Preferably, film recording is the video recording of IT O＆Ms operation display；And/or application is operation and maintenance tools.
Second aspect, an embodiment of the present invention provides a kind of screen content analytical equipment, device includes：
Identification module, for being based on application image model library, active window region and its correspondence in recognition screen image
Application；
Region division module, for the application image model based on the application, the identification module is identified described in
Active window region division is one or more text areas and non-textual area；And
Text region module for identifying the word in the text area of the division module division, obtains character string
Text.
The third aspect, an embodiment of the present invention provides a kind of computing device, including：It is at least one processor, at least one
Memory and computer program instructions stored in memory, are realized when computer program instructions are executed by processor
Such as the method for first aspect in the above embodiment.
Fourth aspect, an embodiment of the present invention provides a kind of computer readable storage mediums, are stored thereon with computer journey
Sequence instructs, and is realized when computer program instructions are executed by processor such as the method for first aspect in the above embodiment.
Screen content analysis method provided in an embodiment of the present invention, device, equipment and medium improve O＆M operation
Accuracy improves the audit efficiency of O＆M operation.
Description of the drawings
It in order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to required in the embodiment of the present invention
The attached drawing used is briefly described, for those of ordinary skill in the art, in the premise not made the creative labor
Under, it can also be obtained according to these attached drawings other attached drawings.
Fig. 1 shows the schematic flow chart of the screen content analysis method of one embodiment of the invention.
Fig. 2 shows an Application Scenarios-Examples of the region division in one embodiment of the invention；
Fig. 3 shows the another application Sample Scenario of the region division in one embodiment of the invention；
Fig. 4 shows the another application Sample Scenario of the region division in one embodiment of the invention；
Fig. 5 shows the schematic flow chart of the screen content analysis method of one embodiment of the invention.
Fig. 6 shows the schematic diagram for extracting non-duplicate frame in film recording of one embodiment of the invention.
Fig. 7 shows the schematic diagram for extracting non-duplicate frame in film recording of one embodiment of the invention.
Fig. 8 shows the schematic block diagram of the screen content analytical equipment of one embodiment of the invention.
Fig. 9 shows the schematic diagram of the computing device of one embodiment of the invention.
Specific embodiment
The feature and exemplary embodiment of various aspects of the invention is described more fully below, in order to make the mesh of the present invention
, technical solution and advantage be more clearly understood, with reference to the accompanying drawings and embodiments, the present invention is further retouched in detail
It states.It should be understood that specific embodiment described herein is only configured to explain the present invention, it is not configured as limiting the present invention.
To those skilled in the art, the present invention can be real in the case of some details in not needing to these details
It applies.The description of embodiment is used for the purpose of by showing that the example of the present invention is better understood from the present invention to provide below.
It should be noted that herein, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, without necessarily requiring or implying between these entities or operation
There are any actual relationship or orders.Moreover, term " comprising ", "comprising" or its any other variant are intended to
Cover non-exclusive inclusion, so that process, method, article or equipment including a series of elements not only include that
A little elements, but also including other elements that are not explicitly listed or further include for this process, method, article or
The intrinsic element of person's equipment.In the absence of more restrictions, the element limited by sentence " including ... ", is not arranged
Except also there are other identical elements in the process, method, article or apparatus that includes the element.
The embodiment of the present invention operates O＆M video recording and carries out differential screening identification frame by frame, based on priority algorithm in video recording
Repeating frame carry out efficient duplicate removal；IT operation and maintenance tools iconic models are modeled simultaneously, to improve the standard of O＆M operation identification
True property.It realizes the content recognition of IT O＆Ms video recording, promotes audit efficiency.
In order to promote recognition efficiency and identification accuracy, content recognition rather than for the national games is carried out according to operation and maintenance tools feature
Tie up desktop picture identification.The present invention carries out image modeling to operation and maintenance tools, and each step is carried out specifically below in conjunction with attached drawing
It is bright.
Fig. 1 shows the schematic flow chart of the screen content analysis method of one embodiment of the invention.
Can be based on application image model library in the step s 100, active window region in recognition screen image and its
Corresponding application.
Wherein, application can be operation and maintenance tools.Tool information can include operation and maintenance tools title, tool major font, work
Has window region range etc..For example, obtaining identification region by the operation and maintenance tools model built, identification region, which is set to, currently should
Active window region, anchor window current location, and window name is coordinated to identify current process；Pass through the tool of structure
Model promotes the contrast of main text area, inhibits non-textual area's image, for example relative to menu area, status bar etc., promote text
The contrast of local area.
In step s 200 can the application image model based on application, by active window region division be one or more
A text area and non-textual area.
Wherein, in the case that known to the window interface pattern of application, can the window interface pattern based on application will be living
Dynamic window area is divided into one or more text areas and non-textual area；
In the case where the window interface pattern of application is unknown, the inherent feature based on text and picture is by active window
Region division is one or more text areas and non-textual area, and according to described in division result updates application image model
Window interface pattern.
For example, the image for known applications type uses document model matching way, quickly to content territorial classification；
The image for not providing application type is intended that adaptive approach is taken to classify, and updates and has document model.Wherein, adaptively
Image-region classification can be based on text and picture inherent feature, picture content areas type is divided into text area and picture
Two class of area.
For example, for the image of known applications type, quickly in content territorial classification, it is special operation and maintenance tools can be combined
Different layout partition functionality block, business characteristic in binding function block, establishes the content recognition rule for block.It is divided into text
Local area and non-textual area, wherein text area are usually mainly the region that text information is dominant, based on preset document model, and
It is usually pure color to know its background, such as the white of text editing type or the black of command window；It usually visually maps in non-textual area
As area, such as the menu bar of tool, dividing column, status bar.
It is illustrated with general purpose O racle access tool PLSQL Developer, is typically divided into menu area, text input
Region, results area etc., as shown in Figure 2.The feature and recognition rule in each region are as follows：
Text input area：Pay close attention to DDL and DML statement, found command feature according to operational order, as select,
The operations such as update, are not concerned with the discrimination of notes content, then coordinate dictionary select, updte keyword configuration carry out in
Hold identification.
Results area：According to the table features of results area, each column field type is all consistent, therefore by row content
Identification, identification process add in regular expressions judgement, be so classified as the canonicals such as telephone number, email address, ID card No. into
Row identification.
Menu area：Menu area word content predefines menu matrix, such as a in advance according to there is no variations11…a1n
Represent the button under File menu, a21…a2nThe button under " engineering " menu is represented, and so on.
Similar situation is illustrated with ftp client access tool FlashFXp, as shown in figure 3, being primarily upon operating area
And results area：
Operating area：The current directory title of concern, and directory name is made of English and Chinese character, coordinates font word
Number fixation, and then carry out Text region.
Tool results area：As shown in figure 4, it predominantly connects, disconnect, exchange files information, according to the two-way knowledge of columns and rows
Not, first row is all [*], is the code of digital Age table information after row [*], this tool information code is fixed, identification
Afterwards, the character string information that can be identified according to default fix information code error correction.
In the screen content analysis method of the embodiment of the present invention, it may be determined that with apply corresponding business scenario, and according to
Business scenario handles the text identified.
Since operation and maintenance tools and business scenario are associated, each operation and maintenance tools are generally only applicable to a kind of business
Scene, if plsql tools are mainly Oracle maintenance access scenes, FlashFXP tools are to carry out file for ftp server
Upload and down operation.
Therefore, by the text classification identified can be one or more parts according to business scenario.
For example, identification process can the text information that arrives of automatic Classification and Identification, in database identification content, host identification
Hold, operation system identifies content etc..
The word of text area is identified in step S300, obtains character string text.
According to one embodiment of the invention, step S300 can include：
To text area into every trade cutting, line of text image is obtained；
Word segmentation is carried out based on separator, obtains word or expression image；
Character cutting is carried out based on grammer and empirical model, obtains character picture；
The feature vector of calculating character image；
The corresponding character of feature based vector identification character picture or word；And
With reference to word segmentation and row cutting as a result, being combined to obtain character string text to character or word.
According to operation and maintenance tools feature, identification process by including extract text image row data (row cutting), based on separation
The word or expression of symbol extracts (word segmentation) and extracts (character cutting) based on the character of grammer and empirical model, for segmentation
The character gone out calculates feature vector (feature extraction), is input to the corresponding character of identification output or word, bluebeard compound in grader and cuts
Point and row cutting result (word combination), final character string text is exported based on dictionary library combination.
It, can be based on the word in preset one or more language identification text areas in the embodiment of the present invention.
Since the language category being likely to occur in actual tool mainly has simplified form of Chinese Character and English, at text identification
The language category plan of reason is defined to this two class, and is respectively Chinese and English creates corresponding feature database, and according to menu bar,
The intrinsic region of the tools such as editing area, status bar carries out Text region.Word after identification can be stored in big data, according to
The analysis ability of big data calculates word frequency, optimizes defined dictionary.
Fig. 5 shows the schematic flow chart of the screen content analysis method of one embodiment of the invention.For known means
The image of model uses document model matching way, quickly to content region division, is associated business, feature extraction, orientation
Identification；The image for not providing application type is intended that adaptive approach is taken to classify, and updates and has tool model.Wherein, certainly
Adapt to image-region classification can be based on text and picture inherent feature, by picture content areas type be divided into text area and
Two class of picture region.
In conclusion can building for operation and maintenance tools be carried out according to specific O＆M environment, specific operation and maintenance tools, service feature
Mould.
Below using operation and maintenance tools PLSQL Developer as specific example, the side of screen content analysis of the present invention is explained
Method.
1) image that need to be identified is read, identification region is obtained by the operation and maintenance tools model built, it is assumed that recognize
Current application active window is Oracle access tool PLSQL Developer.
2) by image, the relative position of current active window is calculated, is such as (33,58) (33,88).
3) the PLSQL Developer Accessorial Tools Storages contrast in relative position is promoted by the tool model of structure, is inhibited
Non-textual area's image.
4) PLSQL Developer are divided into menu area, text input area, results area etc., each region
Feature and recognition rule are as follows：Text input area：DDL and DML statement are paid close attention to, is found command feature according to operational order, such as
The operations such as select, update, are not concerned with the discrimination of notes content, then coordinate matching for dictionary select, updte keyword
Put carry out content recognition；Results area：According to the table features of results area, each column field type is all consistent, therefore is pressed
Row content recognition, identification process add in the judgement of regular expressions, are so classified as telephone number, email address, ID card No. etc.
Canonical is identified；Menu area：Menu area word content is according to there is no variation, in advance predefined menu matrix progress
Content recognition.
5) word that PLSQL Developer tools recognize is deposited into the table that action type is database, literary name
Section is frame sequence, window name, menu content, editing area content, fruiting area content, state area content etc..
Before to operation and maintenance tools image modeling, in order to reduce the analyzing throughput to screen picture, it can be based on preferential
Algorithm carries out efficient duplicate removal to the repeating frame in film recording, extracts the non-duplicate frame in film recording.
According to one embodiment of the invention, the step of extracting the non-duplicate frame in film recording, can include：
Since current non-duplicate frame, with the initial sample interval of setting, cyclic samples are carried out, are taken out twice until adjacent
The frame number of the frame of sample is less than or equal to predetermined threshold, using it is adjacent sample twice in the frame sampled of forward direction as next
Non-duplicate frame, wherein, cyclic samples include：
In response to finding the frame with current non-duplicate frame dissmilarity in sampling in forward direction, will halve in the sampling interval, and hold
The backward sampling of row；
In response to finding the frame similar to current non-duplicate frame into sampling rear, will halve in the sampling interval, and perform
Forward direction is sampled.
For example, in priority algorithm, convergence step is defined, i.e., is received by the scanning result of adjacent area frame picture
Holding back property judges, such as thinks scanning recognition result when the scanning recognition result degree of overlapping of adjacent area reaches certain standard
Convergence；Define Sampling Strategies, i.e., it, can be according to pumping when selecting next frame picture scanning after a frame picture scanning is restrained
Sample algorithm obtains the next frame sampled images of maximal efficiency, accomplishes that information is reduced to greatest extent in the case of not losing and repeats spy
Levy the scanning of frame.Sampling Strategies are sampled including preceding to sampling with backward, and the calculating cyclic process of Sampling Strategies is as follows each time：
After a frame image is identified as convergence, by the preceding sequence number to policy calculation next frame, such as the scanning of the frame
The recognition result similarity of recognition result and former frame then illustrates that the frame belongs to non-heavy frame, then carries out plan backward outside convergence domain
Approximation is calculated, only when the forward direction policy selection frame selected is non-heavy frame, and backward policy selection frame attach most importance to frame when, this is preceding to plan
Slightly selection frame is next scanning frame.
When being sampled to screen picture, can use preceding to sampling and the method to move in circles of sampling backward.
If S points represent search starting point, E points represent search target.If the distance between S points and E points are R, R=R1+R2,
It is roundlet of the center of circle using R2 as radius for the roundlet of radius and using E to draw using S as the R1 in the center of circle.The big area of a circle：S1=π R2=π
(R1+R2)2, the small area of a circle：S2=π R12+πR22=π (R1+R2)2-2πR1R2。
Great circle represents the possible search range of unidirectional search, and two roundlets represent the search range of certain bidirectional research, holds
Easily prove two roundlets area and must be smaller than great circle.Because 2 π R1R2, S1=S2+2 π R1R2 of S2=S1-.S2
Value be less than or equal to S1, therefore the small area of a circle must be smaller than great circle.
So the range of bidirectional research is less than the range of unidirectional search, therefore, the embodiment of the present invention using it is preceding to sampling with
The method that backward sampling combines is sampled film recording.
If the distance between S points and E points are constant R, wherein a roundlet divides radius for X, another small radius of circle is R-X, two
The area of a roundlet is S.
S=π [X2+(R-X)2]；S=2 π [(X-R/2)2+R2\4].Because (X-R/2) 2>=0, so working as X-R/2=0
When, Smin=2 π R24=π R2\2.X=R/2, during two roundlet area equations, i.e. when X points are in intermediate, the two small areas of a circle are most
It is small.Therefore it is preceding as small as possible to sampling and the sample range sampled backward in order to make, each sampling interval can be halved.
Therefore, priority algorithm duplicate removal frame step can include：
1) video data of N minutes is obtained, and calculates the acquisition frequency, it is assumed that the image frame grabber frequency is M seconds
2) the frame image at 0s and Ms is directly acquired
3) subtraction is carried out to the pixel of two frame images
4) if result is 0, then the frame image between 0s and Ms belongs to repeating frame, without content recognition
5) continue, obtain the frame picture at M*2s and carry out subtraction operation with the frame picture at Ms
6) if result is 0, then the frame image at Ms and 2Ms belongs to repeating frame, without content recognition
7) if frame image result of calculation at Ms and 2Ms is not 0, then obtain (M+2M) frame image and Ms at 2s
The image at place is corresponded to
8) when such as Ms is different with (M+2M) 2s, continues rebound and compared at 5M 4s with the frame image of Ms
9) as frame image is identical, then the frame image for obtaining last time comparison carries out content recognition
10) and so on, continue getting frame picture and compared.
Fig. 6-Fig. 7 shows the schematic diagram for extracting non-duplicate frame in film recording of one embodiment of the invention.
As shown in fig. 6, by the sampling interval for 2s, to be spaced the frame image of 2s and current frame image comparison, it is such as different,
Toward rebound 1s, compared again with current frame image, and so on, when sending out identical, the frame picture of last time comparison is taken to carry out content knowledge
Not.
As shown in fig. 7, the frame image of interval 2s and current frame image comparison, such as identical, toward front jumping 2s, with present frame figure
As comparing again, and so on, when finding different, in 2s, toward during rebound, when finding identical, take the frame image of last time comparison
Carry out content recognition.
According to one embodiment of the invention, this method further includes：The frame sequence corresponding to screen picture is recorded in database table
Number and the character string text that is identified in screen picture.
It can be corresponding by recording film recording using the screen picture after sampling as the input of operation and maintenance tools model
Frame number corresponds the character string text identified in film recording and the frame number of film recording.
Film recording mentioned above can be the video recording of IT O＆Ms operation display.By the above method, can be primarily based on
Priority algorithm is realized and the differential screening frame by frame that IT O＆Ms operate is identified, obtains content to be identified；By to IT O＆M works
Tool modeling improves the accuracy of O＆M operation identification, and solving audit operation person can not be based on the operation content in film recording
The problem of carrying out retrieval audit can realize the content recognition of IT O＆Ms operation video recording and the retrieval audit of Video content.
The screen content analysis that the screen content analysis method of the embodiment of the present invention can provide through the embodiment of the present invention
Device is realized.Fig. 8 is shown according to the present invention with the schematic block diagram of the screen content analytical equipment of embodiment.Such as
Shown in Fig. 8, which includes：Identification module 510, region division module 520 and Text region module
530。
Identification module 510 can be based on application image model library, active window region in recognition screen image and its right
The application answered.
Region division module 520 can the application image model based on application, the active window that identification module 510 is identified
Mouth region domain is divided into one or more text areas and non-textual area.
Text region module 530 can obtain character with the word in the text area of the division of identification region division module 520
Illustration and text juxtaposed setting sheet.
Wherein, region division module 520 can include the first division unit and the second division unit.
First division unit can be in the case that known to the window interface pattern of application, the window interface based on application
Active window region division is one or more text areas and non-textual area by pattern.
Second division unit can be in the case where the window interface pattern of application be unknown, consolidating based on text and picture
It by active window region division is one or more text areas and non-textual area, and should according to division result update to have feature
With the window interface pattern described in iconic model.
Text region module can include row cutting unit, word segmentation unit, character cutting unit and computing unit.
Wherein, row cutting unit can obtain line of text image to text area into every trade cutting.
Word segmentation unit can be based on separator and carry out word segmentation, obtain word or expression image.
Character cutting unit can be based on grammer and empirical model carries out character cutting, obtain character picture.
Computing unit can be with the feature vector of calculating character image, so that feature based vector identification character picture corresponds to
Character or word.
Text region module can also include assembled unit, can combine word segmentation and row cutting as a result, to character
Or word is combined to obtain character string text.
According to one embodiment of the invention, which can also include：Associated services module.Association
Business module can be determined with applying corresponding business scenario, to be handled according to business scenario the text identified.
According to one embodiment of the invention, which can also include：Extraction module can carry
The non-duplicate frame in film recording is taken, so that the screen picture that identification module 510 identifies is the screen picture of non-repeating frame.
According to one embodiment of the invention, extraction module can include sampling unit, and sampling unit can be from current non-duplicate
Frame starts, and with the initial sample interval of setting, carries out cyclic samples, be less than until the frame number of the adjacent frame sampled twice or
Equal to predetermined threshold, using it is adjacent sample twice in the frame sampled of forward direction as next non-duplicate frame, wherein, cycle is taken out
Sample includes：In response to finding the frame with current non-duplicate frame dissmilarity in sampling in forward direction, will halve in the sampling interval, and hold
The backward sampling of row；In response to finding the frame similar to current non-duplicate frame into sampling rear, will halve in the sampling interval, and
To sampling before performing.
By above device, priority algorithm can be based on, realizes and the differential screening frame by frame that IT O＆Ms operate is identified, obtain
Take content to be identified；By being modeled to IT operation and maintenance tools, the accuracy of O＆M operation identification is improved, solves audit operation
Member can not carry out the problem of retrieval audit based on the operation content in film recording, can realize the interior of IT O＆Ms operation video recording
Hold identification and the retrieval of Video content is audited.
In addition, the screen content analysis method with reference to Fig. 9 embodiment of the present invention stated can be realized by computing device.
Fig. 9 shows the hardware architecture diagram of computing device provided in an embodiment of the present invention.
Computing device can include processor 601 and be stored with the memory 602 of computer program instructions.Specifically,
Above-mentioned processor 601 can include central processing unit (CPU) or specific integrated circuit (Application Specific
Integrated Circuit, ASIC) or may be configured to implement the embodiment of the present invention one or more integrate electricity
Road.
Memory 602 can include the mass storage for data or instruction.For example it is unrestricted, storage
Device 602 may include hard disk drive (Hard Disk Drive, HDD), floppy disk, flash memory, CD, magneto-optic disk, tape
Or the group of universal serial bus (Universal Serial Bus, USB) driver or two or more the above
It closes.In a suitable case, memory 602 may include the medium of removable or non-removable (or fixed).In suitable situation
Under, memory 602 can be inside or outside data processing equipment.In a particular embodiment, memory 602 is non-volatile
Solid-state memory.In a particular embodiment, memory 602 includes read-only memory (ROM).In a suitable case, the ROM
Can be the ROM of masked edit program, programming ROM (PROM), erasable PROM (EPROM), electric erasable PROM (EEPROM),
Electrically-alterable ROM (EAROM) or the combination of flash memory or two or more the above.
Processor 601 is by reading and performing the computer program instructions stored in memory 602, to realize above-mentioned reality
Apply any one screen content analysis method in example.
In one example, computing device may also include communication interface 603 and bus 610.Wherein, as shown in figure 9, place
Reason device 601, memory 602, communication interface 603 are connected by bus 610 and complete mutual communication.
Communication interface 603 is mainly used for realizing in the embodiment of the present invention between each module, device, unit and/or equipment
Communication.
Bus 610 includes hardware, software or both, and the component of computing device is coupled to each other together.For example and
It is unrestricted, bus may include accelerated graphics port (AGP) or other graphics bus, enhancing Industry Standard Architecture (EISA) bus,
Front Side Bus (FSB), super transmission (HT) interconnection, the interconnection of Industry Standard Architecture (ISA) bus, infinite bandwidth, low pin count
(LPC) bus, memory bus, micro- channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express
(PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association part (VLB) bus or other conjunctions
The combination of suitable bus or two or more the above.In a suitable case, bus 610 may include one or more
A bus.Although specific bus has been described and illustrated in the embodiment of the present invention, the present invention considers any suitable bus or mutual
Even.
In addition, with reference to the screen content analysis method in above-described embodiment, the embodiment of the present invention can provide a kind of computer
Readable storage medium storing program for executing is realized.Computer program instructions are stored on the computer readable storage medium；The computer program refers to
Enable any one the screen content analysis method realized when being executed by processor in above-described embodiment.
In conclusion the present invention solves audit operation person and can not carry out retrieval audit based on the operation content in video recording
Problem, realize the content recognition of IT O＆Ms operation video recording and Video content retrieval audit, while to IT operation and maintenance tools image moulds
Type is modeled, and improves the accuracy of O＆M operation identification, promotes IT O＆Ms operation audit efficiency.
It should be clear that the invention is not limited in specific configuration described above and shown in figure and processing.
For brevity, it is omitted here the detailed description to known method.In the above-described embodiments, it has been described and illustrated several
Specific step is as example.But procedure of the invention is not limited to described and illustrated specific steps, ability
The technical staff in domain can be variously modified after the spirit for understanding the present invention, modification and addition or change the step it
Between sequence.
Structures described above frame functional block shown in figure can be implemented as hardware, software, firmware or their group
It closes.When realizing in hardware, it may, for example, be electronic circuit, application-specific integrated circuit (ASIC), appropriate firmware, insert
Part, function card etc..When being realized with software mode, element of the invention be used to perform needed for task program or
Code segment.Either code segment can be stored in machine readable media program or the data-signal by being carried in carrier wave exists
Transmission medium or communication links are sent." machine readable media " can include being capable of any Jie of storage or transmission information
Matter.The example of machine readable media include electronic circuit, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM),
Floppy disk, CD-ROM, CD, hard disk, fiber medium, radio frequency (RF) link, etc..Code segment can be via such as internet, interior
The computer network of networking etc. is downloaded.
It should also be noted that, the exemplary embodiment referred in the present invention, is retouched based on a series of step or device
State certain methods or system.But the present invention is not limited to the sequence of above-mentioned steps, that is to say, that can be according in embodiment
The sequence referred to performs step, may also be distinct from that the sequence in embodiment or several steps are performed simultaneously.
The above description is merely a specific embodiment, and those skilled in the art can be understood that
It arrives, for convenience of description and succinctly, the specific work process of the system of foregoing description, module and unit can refer to aforementioned
Corresponding process in embodiment of the method, details are not described herein.It should be understood that protection scope of the present invention is not limited thereto, it is any
Those familiar with the art in the technical scope disclosed by the present invention, can readily occur in various equivalent modifications or
It replaces, these modifications or substitutions should be covered by the protection scope of the present invention.
Claims (13)
1. a kind of screen content analysis method, which is characterized in that the method includes：
Based on application image model library, active window region and its corresponding application in recognition screen image；
The active window region division is one or more text areas and Fei Wen by the application image model based on the application
Local area；
It identifies the word in the text area, obtains character string text.
2. according to the method described in claim 1, it is characterized in that, the application image model based on application is by the activity
Window area is divided into the step of one or more text areas and non-textual area and includes：
In the case that known to the window interface pattern of the application, based on the window interface pattern of the application by the activity
Window area is divided into one or more text areas and non-textual area；And/or
In the case where the window interface pattern of the application is unknown, the inherent feature based on text and picture is by the active window
Mouth region domain is divided into one or more text areas and non-textual area, and is updated in the application image model according to division result
The window interface pattern of record.
3. according to the method described in claim 1, it is characterized in that, the method further includes：
It determines and the corresponding business scenario of the application；
The text identified is handled according to the business scenario.
4. according to the method described in claim 3, it is characterized in that, it is described according to the business scenario to the text that is identified into
The step of row processing, includes：
By the text classification identified it is one or more parts according to the business scenario.
5. according to the method described in claim 1, it is characterized in that, the word in the identification text area obtains character string
The step of text, includes：
To the text area into every trade cutting, line of text image is obtained；
Word segmentation is carried out based on separator, obtains word or expression image；
Character cutting is carried out based on grammer and empirical model, obtains character picture；
Calculate the feature vector of the character picture；
The corresponding character of the character picture or word are identified based on described eigenvector.
6. according to the method described in claim 5, it is characterized in that, the word in the identification text area obtains character string
The step of text, further includes：
With reference to word segmentation and row cutting as a result, being combined to obtain the character string text to the character or word.
7. according to the method described in claim 1, it is characterized in that, the method further includes：
Based on the word in text area described in preset one or more language identifications.
8. according to the method described in claim 1, it is characterized in that, the method further includes：
The non-duplicate frame in film recording is extracted, the screen picture is the screen picture of the non-duplicate frame.
9. according to the method described in claim 8, it is characterized in that, it is described extraction film recording in non-duplicate frame the step of wrap
It includes：
Since current non-duplicate frame, with the initial sample interval of setting, cyclic samples are carried out, until the adjacent frame sampled twice
Frame number be less than or equal to predetermined threshold, using it is described it is adjacent sample twice in the frame sampled of forward direction as next non-heavy
Multi-frame,
The cyclic samples include：
In response to finding the frame with current non-duplicate frame dissmilarity in sampling in forward direction, will halve in the sampling interval, and after execution
To sampling；
In response to finding the frame similar to current non-duplicate frame into sampling rear, will halve in the sampling interval, and before performing to
Sampling.
10. according to the method described in claim 8, it is characterized in that, the method further includes：
The frame number corresponding to screen picture and the character string text identified in the screen picture are recorded in database table.
11. a kind of screen content analytical equipment, which is characterized in that described device includes：
Identification module, for being based on application image model library, active window region in recognition screen image and its it is corresponding should
With；
Region division module, for the application image model based on the application, the activity that the identification module is identified
Window area is divided into one or more text areas and non-textual area；And
Text region module for identifying the word in the text area of the division module division, obtains character string text.
12. a kind of computing device, which is characterized in that including：At least one processor, at least one processor and it is stored in institute
The computer program instructions in memory are stated, realize that right such as will when the computer program instructions are performed by the processor
Seek 1-10 any one of them methods.
13. a kind of computer readable storage medium, is stored thereon with computer program instructions, which is characterized in that when the calculating
The method as described in any one of claim 1-10 is realized when machine program instruction is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711423613.6A CN108229481B (en) | 2017-12-25 | 2017-12-25 | Screen content analysis method and device, computing equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711423613.6A CN108229481B (en) | 2017-12-25 | 2017-12-25 | Screen content analysis method and device, computing equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108229481A true CN108229481A (en) | 2018-06-29 |
| CN108229481B CN108229481B (en) | 2020-09-11 |
Family
ID=62648634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711423613.6A Active CN108229481B (en) | 2017-12-25 | 2017-12-25 | Screen content analysis method and device, computing equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108229481B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109801161A (en) * | 2019-03-13 | 2019-05-24 | 上海诚数信息科技有限公司 | Intelligent credit and authentification of message system and method |
| CN110314375A (en) * | 2019-06-25 | 2019-10-11 | 努比亚技术有限公司 | A kind of method for recording of scene of game, terminal and computer readable storage medium |
| CN110796140A (en) * | 2019-10-17 | 2020-02-14 | 北京爱数智慧科技有限公司 | Subtitle detection method and device |
| CN112699860A (en) * | 2021-03-24 | 2021-04-23 | 成都新希望金融信息有限公司 | Method for automatically extracting and sorting effective information in personal tax APP operation video |
| CN115859278A (en) * | 2023-03-01 | 2023-03-28 | 深圳市科力锐科技有限公司 | Software operation behavior auditing method, system, equipment and storage medium |
| WO2023066047A1 (en) * | 2021-10-22 | 2023-04-27 | 华为技术有限公司 | Method for recognizing text information in image, and device and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1630302A (en) * | 2003-12-17 | 2005-06-22 | 大唐移动通信设备有限公司 | Handset capable of automatically recording characters and images, and method of recording and processing thereof |
| CN1753043A (en) * | 2004-09-20 | 2006-03-29 | 上海迪比特实业有限公司 | Remote radio meter register method |
| CN101827264A (en) * | 2009-03-06 | 2010-09-08 | 刘永 | Hierarchical self-adaptive video frame sampling method |
| US8396964B2 (en) * | 2009-05-13 | 2013-03-12 | International Business Machines Corporation | Computer application analysis |
| CN104461545A (en) * | 2014-12-12 | 2015-03-25 | 百度在线网络技术（北京）有限公司 | Method and device of providing contents in mobile terminals to users |
| CN104657141A (en) * | 2015-02-12 | 2015-05-27 | 金光 | GUI (Graphic User Interface) software wrapper based on computer vision and implementing method thereof |
| CN106445144A (en) * | 2016-09-27 | 2017-02-22 | 宇龙计算机通信科技(深圳)有限公司 | Note taking method, device and terminal |
-
2017
- 2017-12-25 CN CN201711423613.6A patent/CN108229481B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1630302A (en) * | 2003-12-17 | 2005-06-22 | 大唐移动通信设备有限公司 | Handset capable of automatically recording characters and images, and method of recording and processing thereof |
| CN1753043A (en) * | 2004-09-20 | 2006-03-29 | 上海迪比特实业有限公司 | Remote radio meter register method |
| CN101827264A (en) * | 2009-03-06 | 2010-09-08 | 刘永 | Hierarchical self-adaptive video frame sampling method |
| US8396964B2 (en) * | 2009-05-13 | 2013-03-12 | International Business Machines Corporation | Computer application analysis |
| CN104461545A (en) * | 2014-12-12 | 2015-03-25 | 百度在线网络技术（北京）有限公司 | Method and device of providing contents in mobile terminals to users |
| CN104657141A (en) * | 2015-02-12 | 2015-05-27 | 金光 | GUI (Graphic User Interface) software wrapper based on computer vision and implementing method thereof |
| CN106445144A (en) * | 2016-09-27 | 2017-02-22 | 宇龙计算机通信科技(深圳)有限公司 | Note taking method, device and terminal |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109801161A (en) * | 2019-03-13 | 2019-05-24 | 上海诚数信息科技有限公司 | Intelligent credit and authentification of message system and method |
| CN110314375A (en) * | 2019-06-25 | 2019-10-11 | 努比亚技术有限公司 | A kind of method for recording of scene of game, terminal and computer readable storage medium |
| CN110796140A (en) * | 2019-10-17 | 2020-02-14 | 北京爱数智慧科技有限公司 | Subtitle detection method and device |
| CN112699860A (en) * | 2021-03-24 | 2021-04-23 | 成都新希望金融信息有限公司 | Method for automatically extracting and sorting effective information in personal tax APP operation video |
| CN112699860B (en) * | 2021-03-24 | 2021-06-22 | 成都新希望金融信息有限公司 | Method for automatically extracting and sorting effective information in personal tax APP operation video |
| WO2023066047A1 (en) * | 2021-10-22 | 2023-04-27 | 华为技术有限公司 | Method for recognizing text information in image, and device and system |
| CN115859278A (en) * | 2023-03-01 | 2023-03-28 | 深圳市科力锐科技有限公司 | Software operation behavior auditing method, system, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108229481B (en) | 2020-09-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108229481A (en) | Screen content analysis method, device, computing device and storage medium | |
| CN110135411B (en) | Business card recognition method and device | |
| WO2020232879A1 (en) | Risk conduction association map optimization method and apparatus, computer device and storage medium | |
| CN102799890A (en) | Image clustering method | |
| CN109726661B (en) | Image processing method and apparatus, medium, and computing device | |
| US11403766B2 (en) | Method and device for labeling point of interest | |
| CN104778238B (en) | The analysis method and device of a kind of saliency | |
| CN113095434A (en) | Target detection method and device, electronic equipment and storage medium | |
| CN111121797B (en) | Road screening method, device, server and storage medium | |
| CN105446705A (en) | Method and device used for determining configuration file feature | |
| CN104750791A (en) | Image retrieval method and device | |
| CN111753721A (en) | Human body posture recognition method and device | |
| CN107515950A (en) | A kind of image processing method, device, terminal and computer-readable recording medium | |
| CN113110995A (en) | System migration test method and device | |
| CN110796000A (en) | Lip sample generation method and device based on bidirectional LSTM and storage medium | |
| CN109542551A (en) | Application icon display methods, device, equipment and medium | |
| CN115857905A (en) | Code conversion method and device for graphical programming, electronic equipment and storage medium | |
| CN113705650B (en) | Face picture set processing method, device, medium and computing equipment | |
| CN114398315A (en) | Data storage method, system, storage medium and electronic equipment | |
| CN109271982A (en) | Multiple identification region recognition methods, identification terminal and readable storage medium storing program for executing | |
| CN109255214B (en) | Authority configuration method and device | |
| CN112560925A (en) | Complex scene target detection data set construction method and system | |
| CN112801960B (en) | Image processing method and device, storage medium and electronic equipment | |
| CN113688263B (en) | Method, computing device, and storage medium for searching for image | |
| CN103778210A (en) | Method and device for judging specific file type of file to be analyzed |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |