CN111598943A

CN111598943A - Book in-position detection method, device and equipment based on book auxiliary reading equipment

Info

Publication number: CN111598943A
Application number: CN202010295742.7A
Authority: CN
Inventors: 汤琦璇; 王忍宝; 王晓斐
Original assignee: Anhui Toycloud Technology Co Ltd
Current assignee: Anhui Toycloud Technology Co Ltd
Priority date: 2020-04-15
Filing date: 2020-04-15
Publication date: 2020-08-28
Anticipated expiration: 2040-04-15
Also published as: CN111598943B

Abstract

The invention discloses a book in-position detection method, a book in-position detection device and book in-position detection equipment based on book auxiliary reading equipment. The invention has the conception that the book image retrieval is abandoned, the integral abnormal input detection is firstly carried out in the imaging range of the camera, and the comparison and analysis of the image information of the local area are carried out on the captured current image and the image which is not in place on the basis, thereby judging the in-place state of the book. The invention considers a plurality of conditions in practical application of a user, filters out other abnormal conditions such as non-book in-place and the like, and greatly reduces the times of triggering to identify the book by the comparison idea of the local target, thereby effectively reducing the operation amount, solving the problem of time consumption for detection and avoiding the problem of network delay when the server is requested to retrieve the book.

Description

Book in-position detection method, device and equipment based on book auxiliary reading equipment

Technical Field

The invention relates to the field of auxiliary reading equipment for books, in particular to a book in-position detection method, device and equipment based on the auxiliary reading equipment for books.

Background

In the market, aiming at specific user groups, such as young children, low-grade students and the like, auxiliary reading equipment, such as a book drawing reading robot and the like, which uses electronic components such as a camera and the like to assist in detecting the content of an entity book is provided, when the book drawing equipment is used, a user needs to place an entity book in a recognizable area of the auxiliary reading equipment, and the equipment processes and outputs the book drawing content through an intelligent recognition technology. Therefore, the in-position state of the book has a certain influence on the normal use of such products, that is, how to let the book auxiliary reading device "know" that the book to be processed exists in the recognizable area plays a fundamental key role in the subsequent processes of recognizing, retrieving, outputting and the like aiming at the content.

Analysis shows that the existing detection of the book in-place state has the problems of overhigh comprehensive cost, complex interactive operation, repeated process and time consumption and the like, and particularly, the in-place detection process lacks reasonable design and has poor consideration on a plurality of abnormal conditions, so that frequent recognition is caused, and the user experience is poor.

Disclosure of Invention

In view of the above, the present invention provides a method, an apparatus, and a device for detecting book in position based on a book reading aid, and accordingly provides a computer readable storage medium and a computer program product, which solve many disadvantages of the existing book in position detection technology by designing a detection method combining abnormal filtering and image comparison, and provide a reliable basis for further processing such as automatic identification, retrieval, and output of book contents.

The technical scheme adopted by the invention is as follows:

in a first aspect, the present invention provides a book in-position detection method based on a book auxiliary reading device, including:

presetting a standard image of a target area in an imaging range based on a specific marker in a state that a book is not in place;

carrying out abnormal input detection on the whole image of each input frame;

after filtering abnormal input, acquiring a current image of the target area in each frame of image;

and comparing the current image with the standard image to determine the current in-position state of the book.

In one possible implementation manner, the performing abnormal input detection on each input whole frame of image includes:

carrying out edge detection frame by frame;

based on the whole image of each frame after edge detection, the following condition judgment is carried out in sequence: a. whether the number of straight lines in the whole image is four or not is judged; b. whether the four straight lines form a rectangle or not; c. whether the area of the rectangle is smaller than a preset threshold value or not; d. whether the rotation angle of the rectangle is smaller than a preset threshold value or not;

and when the judgment result of any one of the conditions is negative, judging the frame image as abnormal input and filtering the frame image.

In one possible implementation manner, the comparing the current image with the standard image includes:

based on a difference hash algorithm, comparing the similarity of the pixel characteristics of the specific marker in the current image and the standard image;

determining that the book is not in a position for the current image meeting the first similarity standard;

extracting the edge characteristics of the current image by using an OTSU algorithm for the current image which does not meet a first similarity standard;

based on edge morphology, comparing the similarity of the edge features of the specific markers in the current image and the standard image;

and determining the current in-position state of the book according to the second similarity standard.

In one possible implementation manner, the method further includes:

before comparing the current image with the standard image, performing brightness detection on each frame of the current image;

and adjusting the brightness of the current image with abnormal brightness and then comparing the current image with the abnormal brightness.

In one possible implementation manner, the method further includes:

dynamically updating the standard image according to the non-in-position state detected in the in-position detection process;

and detecting the book in-position state based on the updated standard image.

In a second aspect, the present invention provides a book seating detection device based on a book reading aid, comprising:

the standard image presetting module is used for presetting a standard image of a target area in an imaging range based on the specific marker in the state that the book is not in place;

the anomaly detection module is used for carrying out anomaly input detection on the whole image of each input frame;

the current image acquisition module is used for acquiring a current image of the target area in each frame of image after filtering abnormal input;

and the in-position state detection module is used for comparing the current image with the standard image to determine the current in-position state of the book.

In one possible implementation manner, the anomaly detection module includes:

a first edge detection unit for performing edge detection frame by frame;

and the abnormal condition judging unit is used for sequentially judging the following conditions based on the whole image of each frame after the edge detection: a. whether the number of straight lines in the whole image is four or not is judged; b. whether the four straight lines form a rectangle or not; c. whether the area of the rectangle is smaller than a preset threshold value or not; d. whether the rotation angle of the rectangle is smaller than a preset threshold value or not;

and the abnormal filtering unit is used for judging the frame image as abnormal input and filtering the frame image when the judgment result of any condition is negative.

In one possible implementation manner, the in-position state detection module includes:

the pixel characteristic comparison unit is used for comparing the similarity of the pixel characteristics of the specific marker in the current image and the standard image based on a difference hash algorithm;

the positioning state first judging unit is used for determining that the book is positioned in an un-positioning state for the current image meeting the first similarity standard;

the second edge detection unit is used for extracting the edge characteristics of the current image by using an OTSU algorithm for the current image which does not meet the first similarity standard;

the edge feature comparison unit is used for comparing the similarity of the edge features of the specific markers in the current image and the standard image based on edge morphology;

and the in-position state second judging unit is used for determining the current in-position state of the book according to the second similarity standard.

In one possible implementation manner, the apparatus further includes: a brightness detection module;

the brightness detection module specifically comprises:

the brightness detection unit is used for detecting the brightness of each frame of the current image before comparing the current image with the standard image;

and the brightness adjusting unit is used for adjusting the brightness of the current image with abnormal brightness.

In one possible implementation manner, the apparatus further includes:

the standard image updating module is used for dynamically updating the standard image according to the non-in-place state detected in the in-place detection process;

the in-position state detection module is also used for detecting the in-position state of the book based on the updated standard image.

In a third aspect, the present invention provides a book reading aid comprising:

one or more processors, memory which may employ a non-volatile storage medium, and one or more computer programs stored in the memory, the one or more computer programs comprising instructions which, when executed by the apparatus, cause the apparatus to perform the method as in the first aspect or any possible implementation of the first aspect.

In one possible implementation manner, the book auxiliary reading device is a book drawing reader, and a specific marker for detecting the book in-position state is arranged on the book drawing reader.

In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon a computer program which, when run on a computer, causes the computer to perform the method as described in the first aspect or any possible implementation manner of the first aspect.

In a fifth aspect, the present invention also provides a computer program product for performing the method of the first aspect or any possible implementation manner of the first aspect, when the computer program product is executed by a computer.

In a possible design of the fifth aspect, the relevant program related to the product may be stored in whole or in part on a memory packaged with the processor, or may be stored in part or in whole on a storage medium not packaged with the processor.

The invention has the conception that the book image retrieval is abandoned, the integral abnormal input detection is firstly carried out in the imaging range of the camera, and the comparison and analysis of the image information of the local area are carried out on the captured current image and the image which is not in place on the basis, thereby judging the in-place state of the book. The invention considers a plurality of conditions in practical application of a user, filters out other abnormal conditions such as non-book in-place and the like, and greatly reduces the times of triggering to identify and reject the book content by the comparison idea of the local target, thereby effectively reducing the operation amount, solving the problem of time consumption in detection and avoiding the problem of network delay when a server is requested to retrieve the book.

Furthermore, the method combines a difference hash algorithm and an edge detection algorithm based on an OTSU algorithm to perform staged matching analysis on the current image information and the standard image information, and the reliability of the in-place detection result is ensured by a progressive judgment process.

Furthermore, the invention provides a comparison method for dynamically updating the standard, so that the method is closer to different use environments of different users, and further can adaptively detect the book in position in an actual application scene, thereby improving the accuracy of the in-position detection result.

Drawings

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described with reference to the accompanying drawings, in which:

fig. 1 is a flowchart of an embodiment of a book-in-place detection method based on a book-assisted reading device according to the present invention;

fig. 2 is a comparison diagram of the state of the reading apparatus with a book in place according to the present invention;

FIG. 3 is a flow chart of an embodiment of an abnormal input filtering method provided by the present invention;

FIG. 4 is a flowchart illustrating an embodiment of an image feature comparison method according to the present invention;

FIG. 5 is a diagram illustrating a differential hash matrix according to an embodiment of the present invention;

FIG. 6 is a flowchart of an embodiment of an edge detection method provided by the present invention;

fig. 7 is a block diagram of an embodiment of a book-in-position detecting device based on a book-assisted reading apparatus provided by the present invention;

fig. 8 is a block diagram of another embodiment of the book-in-position detecting device based on the auxiliary reading apparatus for books provided by the invention.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative only and should not be construed as limiting the invention.

Before explaining the technical scheme of the invention, firstly, the existing book in-position state detection scheme is explained, and currently, in the field of book auxiliary reading equipment, the book in-position detection is generally carried out through the following three modes: firstly, user intervention is carried out, and book auxiliary reading equipment is triggered to start identification processing through interactive operations such as key pressing, selection and the like; secondly, carrying out physical perception of the in-position state through a hardware mechanism; and thirdly, directly searching the book according to the image captured by the camera, namely identifying the captured image aiming at the book content, and triggering to reject if the content is not identified.

Accordingly, the above approaches also produce respective drawbacks: firstly, the user intervention is frequent, the usability is poor, and the user experience is poor; secondly, the difficulty of hardware design and production is increased, and particularly once the hardware is damaged, the maintenance is more inconvenient, so that the comprehensive cost is increased; and thirdly, searching whether a book exists or not in each frame of image captured by the camera continuously needs to be carried out, so that frequent request and repeated operation are needed, the background processing pressure and risk are increased, and the calculation process is complex and time-consuming.

In view of the above, the present invention provides a technical concept of comparing information in an input image with information in a standard image based on setting of a specific marker, so that a user is not required to participate in a book state confirmation link, a mechanical detection mechanism is not required to be configured, and many defects of directly identifying a book are particularly avoided.

In combination with the specific embodiment, the present invention provides an embodiment of a book in-position detection method based on a book reading aid, as shown in fig. 1, which may include the following steps:

step S0 is to preset a standard image of the target area within the imaging range based on the specific marker in the state where the book is not in position.

It will be appreciated that the area within the imaging range in which the book is placed when not in place is relatively constant, so that in a development test session or factory setting, an image of the book when not in place can be taken by the camera of the book reading aid in a normal daylight lamp environment as a reference basis for subsequent frame-by-frame comparisons. In practical operation, by combining with the use characteristics of the book reading aid, a plurality of specific markers, such as but not limited to LOGO characters, device image patterns, etc., are set in the imaging range, and a target area, that is, a local imaging area that the book must cover when being placed normally, is defined in advance based on the specific markers, and particularly, the local imaging area should include the specific markers, so that the book must be pressed against the specific markers when being placed normally, and a comparison basis is provided for comparison of subsequent image information. Continuing from the foregoing, in the preset stage, the area image in the global image (which may be denoted as image _ std) may be intercepted as a standard image (which may be denoted as roi _ std) of the target area a, and then relevant image information of roi _ std is obtained, for example, but not limited to, an edge binary image (which may be denoted as Canny _ std) is obtained by Canny operator edge detection on roi _ std, and or a hash sequence (which may be denoted as hash _ std) is obtained by performing hash calculation after roi _ std is scaled, so that in some embodiments, the image information triplets (roi _ std, Canny _ std, hash _ std) may be used for representation. The technical content of the method will be specifically developed later, but it can be described here that the manner of setting the reference standard image for comparison, which is related to the step, is not only shown in the prefabrication step in other embodiments of the invention, but also will be described later, and will not be described again here.

Step S1, an abnormal input detection is performed for each input frame of the entire image.

In the process of analyzing the book in position, the invention finds that the detection of the in-position state in the real use scene actually meets various conditions, particularly various abnormal illegal input conditions can occur in the captured image, if the abnormality cannot be filtered in advance before the in-position detection, the effect of subsequent in-position detection processing can be interfered, and even the rejection of the book content is frequently triggered.

To facilitate an understanding of the context of the design and associated example scenarios of the present invention, FIG. 2 illustrates a comparison of various book placement scenarios: fig. 2(a) shows the situation where no book is placed, which can also be understood as the aforementioned standard image, where the target area a is a local area to be subsequently compared and detected, and since the schematic diagram is based on the model of some auxiliary reading devices (the devices include a hand plate component for placing the book), this area a can also be called a hand plate feature area, and further, fig. 2(a) uses the text "hand plate" as the specific marker in the area a, and the area C shows the whole image range that can be captured by the camera, i.e. an imaging area; FIG. 2(B) shows the book fully and correctly in place, where the pixels in region A are all part of the book, i.e., region B, which is covered by the solid dark book, completely covers region A and the "hand board" word; FIG. 2(c) is a situation where the book is not fully in place and cannot cover the "palm" two; FIG. 2(d) shows a book not fully seated and only partially covering the "palm" word; FIG. 2(e) is a situation where the book is not placed in the correct manner, e.g., the book is rotated at an angle; FIG. 2(f) is a "pseudo-in-place" situation when other objects than books are placed on the reader's hand plate.

It is clear from these six comparison plots that there is a clear difference in imaging of region a when the book is not in place (a) when the book is placed in various forms (b, c, d, e) and when the non-book object is in place (f). Wherein, (e, f) belongs to the above abnormal input condition, if only the imaging difference of the area A is analyzed, it is difficult to distinguish (e, f) from (b, c, d), therefore, the invention proposes that the complete book in-position detection at least comprises two key parts, firstly, the whole image is analyzed to filter the abnormal condition, and after the abnormal filtering is completed, the local area A is detected in-position, so that the book in-position state can be reliably determined. In implementation, after the auxiliary reading device is started, each frame of continuously captured global image is recorded as img _ test, edge detection is performed on the current frame of img _ test, multi-stage abnormality detection is performed according to a certain condition judgment sequence based on each frame of whole image after edge detection, if the frame of whole image belongs to the abnormal input condition, subsequent local feature detection is not performed, in other words, local feature detection is continued only in the case of judging the frame of whole image as normal input, and the current frame of image is judged as abnormal input and filtered out once the frame of whole image does not accord with the condition.

Specifically, reference may be made to the exception entry filtering process in some preferred embodiments illustrated in FIG. 3. As shown in fig. 3, edge detection is first performed on the current frame image when determining whether the current frame image is an abnormal input, and an improved Canny operator edge detection may be adopted in the edge detection algorithm, which will be described in detail later. And obtaining an edge binary image by edge detection, wherein the pixel value of an edge point is 1, and the pixel value of a non-edge point is 0, and then performing cumulative probability Hough transformation on the edge binary image. It should be noted that, in the present embodiment, the cumulative probability hough transform is considered to have higher execution efficiency compared with the standard hough transform and the multi-scale hough transform; then, detecting straight lines existing in the edge binary image through cumulative probability Hough transform, and returning the number of the detected straight lines, wherein the first abnormal judgment condition is defined in some embodiments whether the number of the straight lines is more than or equal to four or not and whether fewer than four are illegal inputs or not based on book characteristics; or determining whether four lines are equal to each other, that is, less than or more than four lines are considered as illegal inputs (of course, it can be supplemented that if it is determined that straight lines exist in the whole image after edge detection, the coordinates of two end points of the straight lines can also be returned, that is, the information of each straight line is represented by a four-element tuple (x, x)₁，y₁，x₂，y₂))。

When the number of the straight lines is determined to be four, a straight line equation can be determined through two points of the straight lines to obtain four straight lines L₁、L₂、L₃、L₄Thus, it can be further determined whether there are four straight linesThe lines form a rectangle, in the practical operation, the investigation of the condition can be disassembled into two sub-conditions, one is that each straight line in the four straight lines can find a unique straight line parallel to the straight line in the other three straight lines; the other is that the included angle between two straight lines of the non-parallel straight line pair is 90 degrees +/-delta theta, and delta theta represents the deviation of the allowed included angle, so that whether the rectangular frame of the book-like structure is formed or not can be judged.

Then, the side length of four sides of the rectangular frame can be recorded as l₁、l₂、l₃、l₄(ideally, /)₁＝l₃，l₂＝l₄) Judging whether the length of the shortest line segment is less than a set length threshold value l, namely min (l)₁，l₂，l₃，l₄) L is less than. Through controlling the size of the threshold value l, illegal inputs which satisfy four straight lines and can form a rectangle but have an excessively small area of the class region can be filtered, namely rectangular objects with an area obviously smaller than that of a book, such as mobile phones, notepaper and the like, are filtered.

Finally, whether the book is abnormal or not can be determined through four vertexes of a rectangular frame formed by four straight lines. Specifically, the point with the smallest ordinate of the four vertex coordinates can be recorded as P, and two straight lines passing through the point calculate that the included angles between the two straight lines and the positive direction of the x axis are respectively theta₁、θ₂，min(θ₁，θ₂) The number of the book rotated can be set to meet the requirement by controlling the threshold value delta theta', so that the problem of reduced subsequent positioning detection effect caused by overlarge rotation angle is avoided.

In the determination process of the above conditions, once any one condition detects that the condition does not meet the standard, an abnormal filtering operation can be executed, and the whole image of the next frame is continuously extracted to continue the implementation process, that is, if all the conditions are met, the whole image is considered to have a legal entity book which is similar to the book outline and is placed in a standard, so that a processing link of judging whether the book is in place can be further executed in sequence. It should be noted that the above determination conditions are only illustrative, and the specific implementation manner of the determination condition of the abnormal input condition may be adjusted accordingly based on different usage scenarios.

And step S2, filtering the abnormal input and then acquiring the current image of the target area in each frame of image.

After the abnormal filtering, the current image of the target area is intercepted from the whole image of each frame, that is, the global image is cut according to the standard consistent with the standard image position size, so that the current imaging (which can be recorded as roi _ test) of the area a is obtained, and thus, the in-position state of the book can be judged by comparing roi _ std with roi _ test.

Before the description of the specific comparison manner, it may be further described that, in other embodiments of the present invention, before the comparison between the current image and the standard image, brightness detection may be performed on each frame of the current image, and the comparison may be performed after brightness adjustment is performed on the current image with abnormal brightness. Specifically, luminance detection is performed on the roi _ test first, for example, in some embodiments, image luminance detection may be performed based on the average deviation, a relative gray level reference refer (for example, 128 may be selected) is selected by calculating a deviation mean and an average deviation of the image, and a deviation mean Da of the image is calculated:

in the formula (1), i is an image pixel index, and N is the number of image pixels. Obtaining gray level interval [0, 255 ] by utilizing gray level histogram statistical method]Number of pixel points Hist [ val ] corresponding to each gray value]Taking the number of pixels corresponding to the gray level val as a weight, and obtaining the average deviation M of the deviation from the reference by weighting calculation_aAs shown in formula (2):

in equation (2), val ∈ [0, 255 [ ]]For image grey levels by comparison D_aAnd M_aThe absolute value of the ratio yields the luminance parameter K,

when K is less than 1, the image brightness is normal; k is not less than 1 and D_aWhen the brightness is higher than 0, the image is too bright; k is not less than 1 and D_aBelow 0, the image is too dark.

Then, if the detection result is abnormal brightness (over-bright or over-dark), histogram equalization processing can be performed on the image with abnormal brightness, so that the link can avoid the influence of the abnormal brightness of the image on subsequent in-position detection.

It should be emphasized that the above brightness detection and adjustment strategy is designed in the preferred embodiment, and is related to and specific to the in-position detection concept of the present invention, and because the present invention does not need to adopt hardware mechanisms for determining the in-position state, but depends on the image difference (specifically, the difference of the marker in the target region in different images) in the same region at different stages for in-position detection, the brightness detection and adjustment process can play a more critical role in the comparison concept of the present invention, so that the comparison effect between two images is more reliable, and the result is more accurate.

And step S3, comparing the current image with the standard image to determine the current in-position state of the book.

Aiming at the step, the invention designs the implementation mode of in-place detection from the aspect of image feature similarity, and can adopt a staged comparison thought to select features based on all pixel points and edge features from the image features for sequential comparison, which is also two elements in the three elements of the standard image obtained in the prefabrication link mentioned in the foregoing, and the final book in-place state detection result is more accurate through the concepts of stage-by-stage screening, filtering and elimination. Before specific expansion, it should be noted that if the standard image is an image subjected to scaling, the image of the current year may be subjected to scaling processing, such as scaling to the same size of 9 × 8, before feature comparison, and it is understood by those skilled in the art that scaling to a smaller size may eliminate the interference of differences caused by different image sizes and scales.

Continuing, regarding step S3, the present invention in some preferred embodiments adopts a staged comparison process as shown in fig. 4, which may include the following steps:

step S31, comparing the similarity of the pixel characteristics of the specific marker in the current image and the standard image based on a difference hash algorithm;

step S32, determining that the book is not in position for the current image which meets the first similarity standard;

step S33, extracting the edge characteristics of the current image by using an OTSU algorithm for the current image which does not conform to the first similarity standard;

step S34, comparing the similarity of the edge characteristics of the specific marker in the current image and the standard image based on edge morphology;

and step S35, determining the current in-position state of the book according to the second similarity standard.

Specifically, the embodiment obtains a hash sequence (which may be referred to as hash _ test) by a difference hash calculation, so as to represent the pixel characteristics of the image. Of course, there are many types of hash algorithms available in the image processing field, such as mean hash (aHash) and perceptual hash (pHash). However, in the design process, the aHash is considered to be greatly influenced by the image mean value, and if the image is corrected or histogram equalization is carried out, the image mean value is influenced, so that the hash value of the aHash is influenced; and pHash obtains a scattered cosine transform coefficient matrix through discrete cosine transform, and has a lower operation speed than aHash although being more robust than aHash by performing aHash on the upper left corner part of the coefficient matrix. Therefore, in some embodiments, the present invention proposes to perform preliminary comparison of pixel feature similarity by using difference hash (dHash) with high precision and fast operation speed, and the algorithm steps can refer to the following steps:

taking the scaled current image as an example, the difference between adjacent pixels in each row of the current image is calculated, and if the left pixel value is greater than the right pixel value, the left pixel value is marked as 1, otherwise, the left pixel value is marked as 0, and an 8 × 8 difference matrix can be obtained, as shown in fig. 5. The difference hash matrix in fig. 5 can also be regarded as a sequence with a length of d 0-d 63 of 64, so that if the difference hash is performed on the standard image and the current image, two sequences with the same length can be obtained, and then the similarity of the two images can be measured by comparing the similarities of the two sequences, for example, the difference between the two images can be selected by comparing the hamming distance, specifically, the hamming distance (which can be denoted as dist) between hash _ test and hash _ std is calculated, and then it is determined whether dist is greater than a set first similarity threshold T0, if the dist is less than or equal to T0, the two images are considered to be similar, that is, it is determined that the book in the current image is not in place, then the subsequent determination of the current frame can be ended, and the next frame is extracted to repeat the above comparison process. If it is greater than T0, the method proceeds to the next comparison stage for the current frame, for example, Canny operator edge detection is performed on the current image to obtain an edge-detected image (which may be referred to as Canny _ test), and the present invention provides an improved edge detection idea in combination with the existing edge detection process as follows:

firstly, noise can be smoothed through Gaussian filtering, and the possibility that a noise point generates a pseudo edge point in edge detection is eliminated; secondly, calculating image gradient and amplitude by using a Sobel operator and carrying out non-maximum suppression; and finally, detecting image edge points by using a double-threshold algorithm and connecting edges. The edge detection improvement idea provided by the invention is developed around the final dual-threshold algorithm, and particularly, in some preferred embodiments, the Canny operator edge detection is improved by changing the traditional mode of manually and fixedly setting high and low dual thresholds into an OTSU iteration method to select the dual thresholds.

The OTSU iterative double-threshold method (the tsu method) is generally used in the field for directly segmenting an image, and the present invention proposes to apply the OTSU iterative double-threshold method to edge detection to improve the rationality of edge detection, so that the detected edge features have better robustness and adaptivity.

The Canny operator dual threshold mechanism aims to acquire more edges through a high threshold and filter more noisy false edges through a low threshold. Thus, the present invention employs OTSUDetermining a high threshold Th_highDetermining the low threshold Th using an iterative threshold_low. Specifically, refer to the flow shown in fig. 6:

first, selecting an initial threshold

Wherein G is_maxIs the maximum value of image gray scale, G_minIs the minimum value of the image gray scale. By T₀Segmenting the image to obtain regions I₁And region I₂，I₁Is a region of high gray value, I₂Is in a low gray value region, is in₁Further using OTSU algorithm to obtain maximum between-class variance to determine Th_highSetting a threshold t to be I₁Dividing the image into foreground and background areas, and counting the number N of pixels in the foreground₁，I₁The total number of pixel points is M, the foreground accounts for I₁The proportion of the total number of the pixel points is α₀，

Similarly, background is given by I₁The proportion of the total number of the pixel points is α₁，

The total average gray scale of I1 is:

μ＝μ₀×α₀+μ₁×α₁formula (6)

The formula is as follows:

η(t)²＝α₀(μ₀-μ)²+α₁(μ₁-μ)²＝α₀α₁(μ₀-μ₁)²formula (7)

In the formula, T ∈ (T)₀，255]Traversing the interval of t, calculating the t value which leads η to be the maximum value, namely the optimal high threshold value Th_high。

To I₂Further processing to determine optimal low threshold value, taking I₂Minimum gray value ofY₁And the maximum gray value Y₂Let T₁＝(Y₁+Y₂) /2, by T₁Segmentation I₂Calculating the gray average value U of the two parts after the division₁And U₂To find a new threshold value T₂＝(U₁+U₂)/2。|T₂-T₁The value of | < T is the iteration termination condition, T is a set parameter, and is generally measured as small as possible, namely T₂-T₁Is close to the threshold value (Th) of the optimal low threshold value (Th)_lowOtherwise, use T₂Value of (a) in place of T₁And continuing the iteration.

Obtaining a canny _ test after the processing, then carrying out similarity comparison of edge features with the canny _ std in the standard image information, if the edge feature similarity meets a preset second similarity threshold, determining that the book is not in place, and extracting the next frame to continue the processing; and finally determining that the book is in place if the edge characteristic detection results are not similar.

Regarding the way of edge feature alignment, reference can be made to, but not limited to, the following examples: after edge detection, an edge binary image canny _ test of the area A is obtained, and the similarity of the edge features of the two images can be compared by adopting a feature matching algorithm based on edge morphology. Specifically, the canny _ test is morphologically operated, and the image obtained by filling the connected region of the edge can be denoted as edge _ test. In the prefabrication link, edge filling can be carried out on the canny _ std to obtain edge _ std, n feature points are selected from the edge _ std, each feature point is composed of a pixel coordinate and a pixel value and is marked as (x)_i，y_iValue), the same is true for edge _ test to be detected (x)_i，y_i) The pixel values of the eight neighborhood positions are value ', so that whether value ' is equal to value exists in the eight neighborhood pixels or not can be compared, and if the value ' is equal to the value, the characteristic of the position in the two images is considered to be the same. It should be noted that, the pixels in the eight-neighborhood range are taken instead of the exact same coordinate position, so as to take into account the possible deviation of the camera. Subsequently, the number of feature points that do not match after the comparison of the n feature points is represented as M, and if M > M (M is a set threshold), edge is consideredThe edge of _ std is different from the edge _ test, that is, the region a of the current frame (the current image) is different from the region a when the book is not in place (the standard image), that is, it is determined that the book is in place. The book can be considered to be in place by controlling the total covering amount of the specific marker in the area A after the book is in place through the selection of the characteristic points, for example, if only the characteristic points of the lower half part of the area A are selected, the book is judged to be in place only by completely covering the specific marker, if only the specific marker is covered by half, the book is considered not to be in place, so that a user can be informed and guided to correctly perform the book placing action in a guiding mode, the invention is not limited, but the invention needs to be emphasized again, and different similarity comparison standards can be set to adapt to different use requirements, so that whether the book can be classified to be in place or not in place can be determined.

Finally, in the invention, considering that the real use environment is not constant and the great probability is different from the environment during factory test, such as illumination brightness, desktop carriers, shadows of other surrounding objects, and the like, in some preferred embodiments, it is proposed that the standard image can be dynamically updated in the in-place detection process, rather than being in-place detected according to the fixed comparison standard provided in the prefabrication link, so that the compatibility and high adaptability of the detection algorithm can be improved. For example, the standard image may be dynamically updated according to the non-in-position state detected in the in-position detection process, and subsequent book in-position state detection may be performed based on the updated standard image. For example, when a user uses the book auxiliary reading device for the first time, the user performs positioning detection according to a preset standard image from the factory, and as a result, the user is determined that the book is not positioned when the images are aligned due to a shadow generated by the pen container on the desktop, in this embodiment, the image information obtained by this detection is updated to the previously set standard image information to form a new alignment standard, and then, when the user repeatedly performs book placement, the user can perform detection by using the updated standard image, so that the positioned book can be successfully detected, and subsequent identification, retrieval, output and the like of the book content can be started. Regarding the timing of dynamic update, the current in-position state of the book may be determined based on the change of the previous and subsequent frames, and the detection criterion may be updated accordingly. The present invention provides an exemplary reference embodiment, for example, a global variable lastState may be set during in-place detection, and lastState is initialized to 0 to indicate the detection result of the previous frame. If the result after detection is that the image is in place, setting lastState to 1, and if the result of detecting the next frame is that the image is not in place, checking whether lastState is 1, if lastState is 1, indicating that the current frame is the image of the book changed from the in-place state to the non-in-place state, then modifying lastState to 0, and using the image information of the current frame as the standard image information, thereby realizing the dynamic update, and ensuring that the information of the standard image can be close to the current real use environment in real time.

Therefore, through the processes of exception filtering, brightness detection, difference hashing, hash sequence comparison, edge feature extraction, edge feature similarity comparison and the like introduced in the foregoing specific embodiment, the book in-place state is continuously detected in multiple stages after the device is turned on, so that the book in-place condition in the use process of the device can be effectively judged, and then the turn-on time for subsequent processing such as identification of book content can be determined, and thus the number of subsequent identification requests and rejection times is reduced to a certain extent.

In summary, the present invention abandons direct retrieval of book contents, and performs overall abnormal input detection in the imaging range of the camera, and on the basis, compares and analyzes the image information of local areas of the captured current image and the image which is not in place, thereby determining the in-place state of the book. The invention considers a plurality of conditions in practical application of a user, filters out other abnormal conditions such as non-book in-place and the like, and greatly reduces the times of triggering to identify and reject the book content by the comparison idea of the local target, thereby effectively reducing the operation amount, solving the problem of time consumption in detection and avoiding the problem of network delay when a server is requested to retrieve the book.

Corresponding to the above embodiments and preferred solutions, the present invention further provides an embodiment of a book-in-place detecting device based on a book reading aid, as shown in fig. 7, which may specifically include the following components:

the standard image presetting module 0 is used for presetting a standard image of a target area in an imaging range based on a specific marker in a state that a book is not in place;

an anomaly detection module 1, configured to perform anomaly input detection on an entire input image of each frame;

the current image acquisition module 2 is used for filtering abnormal input and acquiring a current image of the target area in each frame of image;

and the in-position state detection module 3 is used for comparing the current image with the standard image to determine the current in-position state of the book.

In one possible implementation manner, the anomaly detection module includes:

a first edge detection unit for performing edge detection frame by frame;

and the abnormal condition judging unit is used for sequentially judging the following conditions based on the whole image of each frame after the edge detection: a |, whether the number of straight lines in the whole image is four or not; b. whether the four straight lines form a rectangle or not; c. whether the area of the rectangle is smaller than a preset threshold value or not; d. whether the rotation angle of the rectangle is smaller than a preset threshold value or not;

In one possible implementation manner, as shown in fig. 8, the apparatus further includes: a brightness detection module 21;

the brightness detection module specifically comprises:

In one possible implementation manner, as shown in fig. 8, the apparatus further includes:

the standard image updating module 4 is used for dynamically updating the standard image according to the non-in-place state detected in the in-place detection process;

It should be understood that the book-in-place detection device based on the book-assisted reading apparatus shown in fig. 7 and 8 can be used as a subsystem of an online system, or can be used alone as a question-answering system. Moreover, the division of each component is only a division of a logic function, and all or part of the actual implementation may be integrated into one physical entity or may be physically separated. And these components may all be implemented in software invoked by a processing element; or may be implemented entirely in hardware; and part of the components can be realized in the form of calling by the processing element in software, and part of the components can be realized in the form of hardware. For example, a certain module may be a separate processing element, or may be integrated into a certain chip of the electronic device. Other components are implemented similarly. In addition, all or part of the components can be integrated together or can be independently realized. In implementation, each step of the above method or each component above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.

For example, the above components may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors (DSPs), one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, these components may be integrated together and implemented in the form of a System-On-a-Chip (SOC).

In view of the foregoing examples and their preferred embodiments, it will be appreciated by those skilled in the art that in practice, the invention may be practiced in a variety of embodiments, and that the invention is illustrated schematically in the following vectors:

(1) a book assist reading apparatus, which may comprise:

one or more processors, memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions, which when executed by the apparatus, cause the apparatus to perform the steps/functions of the foregoing embodiments or equivalent implementations.

(2) A readable storage medium, on which a computer program or the above-mentioned apparatus is stored, which, when executed, causes the computer to perform the steps/functions of the above-mentioned embodiments or equivalent implementations.

In the several embodiments provided by the present invention, any function, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in a computer readable storage medium. Based on this understanding, some aspects of the present invention may be embodied in the form of software products, which are described below, or portions thereof, which substantially contribute to the art.

(3) A computer program product (which may include the above apparatus), when run on a book-assisted reading device, causes the device to perform the book-in-position detection method based on the book-assisted reading device of the preceding example or an equivalent embodiment.

From the above description of the embodiments, it is clear to those skilled in the art that all or part of the steps in the above implementation method can be implemented by software plus a necessary general hardware platform.

In the embodiments of the present invention, "at least one" means one or more, "and" a plurality "means two or more. "and/or" describes the association relationship of the associated objects, and means that there can be three relationships, for example, a and/or B, and can mean that a exists alone, a and B exist simultaneously, and B exists alone. Wherein A and B can be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c may represent: a, b, c, a and b, a and c, b and c or a and b and c, wherein a, b and c can be single or multiple.

Those of skill in the art will appreciate that the various modules, elements, and method steps described in the embodiments disclosed in this specification can be implemented as electronic hardware, combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In addition, the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments may be referred to each other. In particular, for embodiments of devices, apparatuses, etc., since they are substantially similar to the method embodiments, reference may be made to some of the descriptions of the method embodiments for their relevant points. The above-described embodiments of devices, apparatuses, etc. are merely illustrative, and modules, units, etc. described as separate components may or may not be physically separate, and may be located in one place or distributed in multiple places, for example, on nodes of a system network. Some or all of the modules and units can be selected according to actual needs to achieve the purpose of the above-mentioned embodiment. Can be understood and carried out by those skilled in the art without inventive effort.

The structure, features and effects of the present invention have been described in detail with reference to the embodiments shown in the drawings, but the above embodiments are merely preferred embodiments of the present invention, and it should be understood that technical features related to the above embodiments and preferred modes thereof can be reasonably combined and configured into various equivalent schemes by those skilled in the art without departing from and changing the design idea and technical effects of the present invention; therefore, the invention is not limited to the embodiments shown in the drawings, and all the modifications and equivalent embodiments that can be made according to the idea of the invention are within the scope of the invention as long as they are not beyond the spirit of the description and the drawings.

Claims

1. A book in-position detection method based on a book auxiliary reading device is characterized by comprising the following steps:

carrying out abnormal input detection on the whole image of each input frame;

2. The book-in-place detection method according to claim 1, wherein said performing abnormal input detection on each input frame of the entire image comprises:

carrying out edge detection frame by frame;

3. The book-in-place detection method of claim 1, wherein the comparing the current image with the standard image comprises:

4. A method of detecting book-in-position according to any one of claims 1 to 3, further comprising:

5. A method of detecting book-in-position according to any one of claims 1 to 3, further comprising:

and detecting the book in-position state based on the updated standard image.

6. A book in-position detection device based on a book auxiliary reading device, comprising:

7. The book-in-position detecting device according to claim 6, wherein the device further comprises: a brightness detection module;

the brightness detection module specifically comprises:

8. The book-in-position detecting device according to claim 6, wherein the device further comprises:

9. A book reading aid, comprising:

one or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs comprising instructions which, when executed by the apparatus, cause the apparatus to perform the method of any of claims 1-5.

10. A book reading aid according to claim 9 wherein the book reading aid is a book drawing reader and wherein a specific marker is provided on the book drawing reader for detecting the book in-position.