WO2018167974A1

WO2018167974A1 - Image processing device, control method, and control program

Info

Publication number: WO2018167974A1
Application number: PCT/JP2017/011039
Authority: WO
Inventors: 雄毅笠原
Original assignee: 株式会社Pfu
Priority date: 2017-03-17
Filing date: 2017-03-17
Publication date: 2018-09-20
Also published as: JP6707178B2; JPWO2018167974A1

Abstract

Provided are an image processing device, a control method, and a control program which make it possible to efficiently store an appropriate image as a proof image. An image processing device has: a storage unit; an image pickup unit for sequentially generating input images by taking images of a meter; a numerical value recognition unit for specifying and compiling numerical values in the meter in the sequentially generated input images, and on the basis of the compilation result, recognizing the numerical values in the meter; a determination unit for determining, for each partial area corresponding to each digit of the numerical values in the meter, whether each partial area is clear or not; and a control unit for selecting, for each digit, an input image in which the partial area corresponding to each digit is determined to be clear, and storing, in the storage unit, at least part of the selected input image as a proof image with the at least part of the selected input image associated with the numerical values recognized by the numerical value recognition unit.

Description

Image processing apparatus, control method, and control program

The present disclosure relates to an image processing device, a control method, and a control program, and more particularly, to an image processing device, a control method, and a control program for recognizing a numerical value in a meter from an image captured by the meter.

In factories, houses, etc., during facility inspection work, workers visually read numerical values indicating the amount of power from a meter (device) such as the amount of power and record it in an inspection book, which is a paper ledger. However, in such a manual operation, an erroneous numerical value is recorded in the inspection book due to human error, and there is a possibility that reworking may occur. In order to solve such problems, in recent years, a technology for automatically recognizing numerical values by a computer from an image obtained by photographing a meter with a camera has been used in equipment inspection work.

An instrument reading device is disclosed that captures a scene including a numerical value indicated by a digital instrument, processes the captured image, and detects a numerical value indicated by the instrument (see Patent Document 1).

Japanese Patent No. 5530752

When the numerical value is automatically recognized by the computer from the image taken by the camera, it is desirable to save the image taken by the camera as an evidence image so that an operator or the like can confirm later. In that case, it is desired that an appropriate image as an evidence image can be efficiently stored.

The purpose of the image processing apparatus, control method, and control program is to enable efficient storage of appropriate images as evidence images.

An image processing apparatus according to an aspect of the present invention specifies a storage unit, an imaging unit that sequentially generates an input image obtained by capturing a meter, and a numerical value in the meter that is reflected in each of the sequentially generated input images. The numerical value recognition unit for recognizing the numerical value in the meter based on the counting result, and whether or not the partial area corresponding to each digit is clear for each digit corresponding to each digit of the numerical value in the meter. For each partial region to be determined, the determination unit selects an input image for which the partial region is determined to be clear, and associates the numerical value recognized by the numerical value recognition unit with at least a part of the selected input image as an evidence image And a control unit stored in the storage unit.

A control method according to an aspect of the present invention is a control method for an image processing apparatus that includes a storage unit and an imaging unit that sequentially generates input images obtained by capturing a meter. The numerical values in the meter reflected in the meter are identified and counted, and the numerical values in the meter are recognized based on the counting results, and each partial area is clear for each partial area corresponding to each digit in the meter. A numerical value that determines whether or not there is an image, and for each digit, selects an input image for which the partial area corresponding to each digit is determined to be clear, and recognizes at least part of the selected input image as an evidence image Are stored in the storage unit in association with each other.

A control program according to an aspect of the present invention is a control program for an image processing apparatus that includes a storage unit and an imaging unit that sequentially generates input images obtained by photographing a meter. The numerical values in the meter reflected in the meter are identified and counted, and the numerical values in the meter are recognized based on the counting results, and each partial area is clear for each partial area corresponding to each digit in the meter. A numerical value that determines whether or not there is an image, and for each digit, selects an input image for which the partial area corresponding to each digit is determined to be clear, and recognizes at least part of the selected input image as an evidence image Are associated with each other and stored in the storage unit.

According to the present embodiment, the image processing apparatus, the control method, and the control program can efficiently store an appropriate image as an evidence image.

The objects and advantages of the invention will be realized and obtained by means of the elements and combinations particularly pointed out in the appended claims. Both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

It is a figure which shows an example of schematic structure of the image processing apparatus 100 according to embodiment. 2 is a diagram illustrating a schematic configuration of a storage device 110 and a CPU 120. FIG. It is a flowchart which shows the example of operation | movement of a whole process. It is a flowchart which shows the example of operation | movement of a partial area | region detection process. It is a figure which shows an example of the input image 500 which image | photographed the meter. It is a figure for demonstrating a partial area | region. It is a figure which shows an example of the data structure of a management table. It is a flowchart which shows the example of operation | movement of a numerical value recognition process. It is a flowchart which shows the example of operation | movement of evidence image preservation | save processing. It is a figure explaining the combination of the evidence image preserve | saved. 6 is a diagram showing a schematic configuration of another processing circuit 230. FIG.

Hereinafter, an image processing apparatus according to an aspect of the present disclosure will be described with reference to the drawings. However, it should be noted that the technical scope of the present disclosure is not limited to the embodiments, and extends to the invention described in the claims and equivalents thereof.

FIG. 1 is a diagram illustrating an example of a schematic configuration of an image processing apparatus 100 according to the embodiment.

The image processing apparatus 100 is a portable information processing apparatus such as a tablet PC, a multi-function mobile phone (so-called smart phone), a portable information terminal, and a notebook PC, and is used by an operator who is the user. The image processing apparatus 100 includes a communication device 101, an input device 102, a display device 103, an imaging device 104, a storage device 110, a CPU (Central Processing Unit) 120, and a processing circuit 130. Hereinafter, each part of the image processing apparatus 100 will be described in detail.

The communication device 101 includes a communication interface circuit including an antenna mainly having a 2.4 GHz band, a 5 GHz band, or the like as a sensitive band. The communication apparatus 101 performs wireless communication with an access point or the like based on an IEEE (The Institute of Electrical and Electronics Electronics, Inc.) 802.11 standard wireless communication system. The communication device 101 transmits / receives data to / from an external server device (not shown) via an access point. The communication apparatus 101 supplies the data received from the server apparatus via the access point to the CPU 120, and transmits the data supplied from the CPU 120 to the server apparatus via the access point. The communication device 101 may be any device that can communicate with an external device. For example, the communication device 101 may communicate with a server device via a base station device (not shown) according to a mobile phone communication method, or may communicate with a server device according to a wired LAN communication method.

The input device 102 has a touch panel type input device, an input device such as a keyboard and a mouse, and an interface circuit that acquires signals from the input device. The input device 102 receives a user input and outputs a signal corresponding to the user input to the CPU 120.

The display device 103 includes a display composed of liquid crystal, organic EL (Electro-Luminescence), and the like, and an interface circuit that outputs image data or various information to the display. The display device 103 is connected to the CPU 120 and displays the image data output from the CPU 120 on a display. Note that the input device 102 and the display device 103 may be integrally configured using a touch panel display.

The imaging device 104 includes a reduction optical system type imaging sensor including an imaging element made up of a CCD (Charge Coupled Device) arranged one-dimensionally or two-dimensionally, and an A / D converter. The imaging device 104 is an example of an imaging unit, and sequentially captures a meter according to an instruction from the CPU 120 (for example, 30 frames / second). The image sensor generates an analog image signal obtained by photographing the meter and outputs the analog image signal to the A / D converter. The A / D converter performs analog-digital conversion on the output analog image signal to sequentially generate digital image data, and outputs the digital image data to the CPU 120. In place of the CCD, an equal magnification optical system type CIS (Contact Image Sensor) provided with an image sensor composed of CMOS (Complementary Metal Metal Oxide Semiconductor) may be used. Hereinafter, digital image data output by the meter captured by the imaging device 104 may be referred to as an input image.

The storage device 110 is an example of a storage unit. The storage device 110 includes a memory device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk and an optical disk. Further, the storage device 110 stores computer programs, databases, tables, and the like used for various processes of the image processing apparatus 100. The computer program may be installed from a computer-readable portable recording medium such as a CD-ROM (compact disk read only memory) or a DVD ROM (digital versatile disk read only memory). The computer program is installed in the storage device 110 using a known setup program or the like. The storage device 110 also stores a management table that manages information related to each input image.

The CPU 120 operates based on a program stored in the storage device 110 in advance. The CPU 120 may be a general purpose processor. Instead of the CPU 120, a DSP (digital signal processor), an LSI (large scale integration), or the like may be used. Instead of the CPU 120, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like may be used.

The CPU 120 is connected to the communication device 101, the input device 102, the display device 103, the imaging device 104, the storage device 110, and the processing circuit 130, and controls these units. The CPU 120 performs data transmission / reception control via the communication device 101, input control of the input device 102, display control of the display device 103, imaging control of the imaging device 104, control of the storage device 110, and the like. Further, the CPU 120 recognizes a numerical value in the meter reflected in the input image generated by the imaging device 104 and stores the evidence image in the storage device 110.

The processing circuit 130 performs predetermined image processing such as correction processing on the input image acquired from the imaging device 104. Note that an LSI, DSP, ASIC, FPGA, or the like may be used as the processing circuit 130.

FIG. 2 is a diagram showing a schematic configuration of the storage device 110 and the CPU 120.

As shown in FIG. 2, the storage device 110 stores programs such as a numerical value recognition program 111, a determination program 112, and a control program 113. Each of these programs is a functional module implemented by software operating on the processor. The CPU 120 functions as the numerical value recognition unit 121, the determination unit 122, and the control unit 123 by reading each program stored in the storage device 110 and operating according to each read program.

FIG. 3 is a flowchart showing an example of the operation of the entire process performed by the image processing apparatus 100.

Hereinafter, an example of the operation of the entire process performed by the image processing apparatus 100 will be described with reference to the flowchart shown in FIG. The operation flow described below is mainly executed by the CPU 120 in cooperation with each element of the image processing apparatus 100 based on a program stored in the storage device 110 in advance.

First, the numerical value recognition unit 121 receives a shooting start instruction when a user inputs a shooting start instruction for instructing the start of shooting with the input device 102 and receives a shooting start instruction signal from the input device 102. (Step S101). When receiving a shooting start instruction, the numerical value recognition unit 121 initializes information used for image processing, sets parameters such as the shooting size and focus of the imaging device 104, and causes the imaging device 104 to take a meter. To generate an input image. The numerical value recognition unit 121 sequentially stores input images sequentially generated by the imaging device 104 in the storage device 110.

Next, the numerical value recognition unit 121 executes a partial area detection process (step S102). In the partial area detection process, the numerical value recognition unit 121 detects a partial area corresponding to each digit of the numerical value in the meter shown in the input image generated by the imaging device 104. Details of the partial area detection processing will be described later.

Next, the numerical value recognition unit 121 determines whether a partial area that can be used in the numerical value recognition process is detected in the partial area detection process (step S103).

When the usable partial area is not extracted in the numerical value recognition process, the numerical value recognition unit 121 returns the process to step S102, and executes the partial area extraction process on the newly generated input image. On the other hand, when a partial area that can be used in the numerical value recognition process is extracted, the numerical value recognition unit 121 executes the numerical value recognition process (step S104). In the numerical value recognition processing, the numerical value recognition unit 121 identifies and counts the numerical values in the meter shown in the sequentially generated input images, and recognizes the numerical values in the meter based on the totaled result. Details of the numerical value recognition processing will be described later.

Next, the numerical value recognition unit 121 determines whether or not the numerical value in the meter has been recognized in the numerical value recognition process (step S105).

When the numerical value in the meter cannot be recognized, the numerical value recognition unit 121 returns the process to step S102, and repeats the processes of steps S102 to S105 for the newly generated input image. On the other hand, when the numerical value in the meter can be recognized, the determination unit 122 and the control unit 123 execute evidence image storage processing (step S106). In the evidence image storage process, the determination unit 122 determines whether each partial area is clear. Further, for each digit, the control unit 123 selects an input image in which the partial area corresponding to each digit is determined to be clear, and uses the image corresponding to the selected input image as an evidence image. The numerical value recognized by 121 is associated and stored in the storage device 110. Details of the evidence image storage process will be described later.

Next, the control unit 123 displays the numerical value recognized by the numerical value recognition unit 121 and / or the evidence image selected by the control unit 123 on the display device 103 (step S107), and ends a series of steps. Further, the control unit 123 may transmit the numerical value recognized by the numerical value recognition unit 121 and / or the evidence image selected by the control unit 123 to the server device via the communication device 101.

FIG. 4 is a flowchart showing an example of the operation of the partial area detection process. The operation flow shown in FIG. 4 is executed in step S102 of the flowchart shown in FIG.

First, the numerical value recognition unit 121 detects a plate frame from the input image (step S201).

FIG. 5A is a diagram showing an example of an input image 500 obtained by photographing a meter (device).

As shown in FIG. 5A, generally, a meter has a black casing 501 and a white plate 502 inside the casing 501. The plate 502 is visible through glass (not shown), and a meter portion 503 on which a numerical value such as the amount of electric power measured by the meter is displayed is disposed on the plate 502. In the meter portion 503, the numerical value is shown in white and the background is shown in black. The numerical value recognition unit 121 detects the outer edge of the plate 502 as a plate frame. In the following description, a meter whose number of numerical values to be measured is four will be described as an example, but any number of numerical values to be measured by the meter may be any number as long as it is two or more.

The numerical value recognition unit 121 is a difference between luminance values or color values (R value, B value, G value) of pixels adjacent to each other in the horizontal and vertical directions in the input image or a plurality of pixels separated from the pixels by a predetermined distance. If the absolute value of exceeds the first threshold, the pixel is extracted as an edge pixel. The numerical value recognition unit 121 extracts a straight line that passes through the vicinity of each extracted edge pixel by using the Hough transform or the least squares method, and two of the extracted straight lines are obtained from four straight lines that are approximately orthogonal to each other. Among the rectangles to be configured, the largest rectangle is detected as a plate frame. Alternatively, the numerical value recognition unit 121 determines whether each extracted edge pixel is connected to other edge pixels, and labels the connected edge pixels as one group. The numerical value recognition unit 121 may detect, as a plate frame, an outer edge of a region surrounded by the largest group among the extracted groups.

Note that the numerical value recognition unit 121 may detect the plate frame using the difference between the color of the housing 501 and the color of the plate 502. The numerical value recognition unit 121 has a luminance value or a color value of each pixel that is less than the second threshold value (shows black), and a pixel adjacent to the pixel on the right side or a pixel separated from the pixel by a predetermined distance on the right side If the color value is greater than or equal to the second threshold (indicating white), that pixel is extracted as the left edge pixel. The second threshold value is set to an intermediate value between the value indicating black and the value indicating white. Similarly, the numerical value recognition unit 121 has a luminance value or a color value of each pixel that is less than the second threshold value and that is adjacent to the pixel on the left side or a pixel that is a predetermined distance away from the pixel on the left side. Is greater than or equal to the second threshold, the pixel is extracted as the right edge pixel. Similarly, the numerical value recognition unit 121 has a luminance value or a color value of each pixel that is less than the second threshold value, and the luminance value or the pixel of the pixel adjacent to the pixel on the lower side or the pixel separated by a predetermined distance from the pixel on the lower side. If the color value is greater than or equal to the second threshold, the pixel is extracted as the upper edge pixel. Similarly, the numerical value recognition unit 121 has a luminance value or a color value of each pixel that is less than the second threshold value and is adjacent to the pixel on the upper side or a pixel that is a predetermined distance above the pixel. Is greater than or equal to the second threshold, the pixel is extracted as the lower edge pixel.

The numerical value recognition unit 121 extracts a straight line connecting the extracted left end edge pixel, right end edge pixel, upper end edge pixel, and lower end edge pixel by using Hough transform or least square method, and is configured from the extracted straight lines. The rectangle to be detected is detected as a plate frame.

Next, the numerical value recognition unit 121 detects a numerical background frame from the detected area within the plate frame (step S202). The numerical value recognition unit 121 detects the outer edge of the meter portion 503 as a numerical value background frame.

The numerical value recognition unit 121 uses a discriminator that has been pre-learned to output the position information of the outer edge of the meter portion 503 when an image showing the plate 502 including the meter portion 503 is input. To detect. This discriminator is pre-learned using a plurality of images obtained by photographing the meter, for example, by deep learning, and stored in the storage device 110 in advance. The numerical value recognition unit 121 detects the numerical value background frame by inputting an image including the detected plate frame to the classifier and acquiring the position information output from the classifier.

Note that the numerical value recognition unit 121 may detect the numerical value background frame based on the edge pixels in the input image, as in the case of detecting the plate frame. The numerical value recognition unit 121 extracts edge pixels from the region including the plate frame of the input image, extracts straight lines passing through the vicinity of each extracted edge pixel, and two of the extracted straight lines are approximately four orthogonal to each other. The largest rectangle among the rectangles composed of straight lines is detected as a numerical background frame. Or the numerical value recognition part 121 may detect the outer edge of the area | region enclosed by the group with the largest area among the groups which each extracted edge pixel connects mutually as a numerical value background frame.

Alternatively, the numerical value recognition unit 121 may detect the plate frame using the difference between the color of the plate 502 and the color of the meter portion 503, as in the case of detecting the plate frame. The numerical value recognition unit 121 has a luminance value or a color value of each pixel equal to or greater than a second threshold value (indicating white), and the luminance value of a pixel adjacent to the pixel on the right side or a pixel separated by a predetermined distance from the pixel to the right side If the color value is less than the second threshold value (indicating black), the pixel is extracted as the left edge pixel. Similarly, the numerical value recognition unit 121 extracts the right edge pixel, the upper edge pixel, and the lower edge pixel. The numerical value recognition unit 121 extracts straight lines that pass through the vicinity of the extracted left end edge pixel, right end edge pixel, upper end edge pixel, and lower end edge pixel by using the Hough transform or the least square method. Is detected as a numeric background frame.

In addition, when the mark 504 indicating the position where the numerical value background frame is present in the meter plate 502 is shown, the numerical value recognition unit 121 detects the mark 504 and is sandwiched between the marks 504 in the horizontal and vertical directions. A numerical background frame may be detected within the region.

Next, the numerical value recognition unit 121 detects a partial region corresponding to each digit of the numerical value in the meter from the detected region in the numerical value background frame (step S203).

FIG. 5B is a diagram for explaining a partial region.

An image 510 shown in FIG. 5B shows a numerical background frame of the meter portion 503 detected from the input image 500. The numerical value recognition unit 121 detects the rectangular areas 511 to 514 including the numerical value values in the numerical value background frame as partial areas.

The numerical value recognition unit 121 is a classifier that has been pre-learned so as to output position information of each rectangular area including each digit value of the numerical value in the meter portion 503 when an image in which the meter portion 503 is reflected is input. Thus, the partial area is detected. This discriminator is pre-learned using a plurality of images obtained by photographing the meter, for example, by deep learning, and stored in the storage device 110 in advance. The numerical value recognition unit 121 detects a partial region by inputting an image including the detected numerical value background frame to a classifier and acquiring position information output from the classifier.

Note that the numerical value recognition unit 121 may detect the numerical value background frame based on the edge pixels in the input image, as in the case of detecting the plate frame. The numerical value recognition unit 121 extracts edge pixels from the area including the numerical value background frame of the input image, extracts straight lines passing through the vicinity of the extracted edge pixels, and two of the extracted straight lines are approximately orthogonal to each other. The largest rectangular area is detected from the four straight lines. Alternatively, the numerical value recognition unit 121 detects a region surrounded by a group having the largest area among the groups in which the extracted edge pixels are connected to each other. The numerical value recognition unit 121 detects a single-digit number from each detected region using a known OCR (Optical Character Recognition) technology, and detects a single-digit number as a partial region. To do.

As shown in FIG. 5B, there is a possibility that the numerical value is not properly captured in the meter portion 503 detected from the input image 500. An image 520 shown in FIG. 5B shows an image in which the entire meter portion 503 is unclear and the difference in luminance value or color value between the numerical value portion and the background portion is small. In the image 520, the rectangular areas 521 to 524 are not detected as partial areas. An image 530 shown in FIG. 5B shows an image in which ambient light such as illumination is reflected on the glass portion covering the front surface of the meter, so that disturbance light is reflected on the entire meter portion 503 and all numerical values cannot be identified. In the image 530, the rectangular areas 531 to 534 are not detected as partial areas. An image 540 shown in FIG. 5B shows an image in which disturbance light is reflected in a part of the meter portion 503 and a numerical value in which disturbance light is reflected cannot be identified. In the image 540, the

rectangular areas

541, 543, and 544 are detected as partial areas, but the rectangular area 542 is not detected as a partial area. An image 550 shown in FIG. 5B shows an image in which a shadow is reflected in a part of the meter portion 503 and a numerical value where the shadow is reflected is erroneously recognized. In the image 550, the rectangular areas 551 to 554 are detected as partial areas.

Next, the numerical value recognition unit 121 assigns a digit number to each detected partial area (step S204). The numerical value recognition unit 121 divides the region in the numerical value background frame equally in the horizontal direction by the number of digits of the numerical value indicated by the meter, and assigns the digit numbers in ascending order from the right region (1 from the rightmost region in order). 2, 3, 4 digit numbers are assigned). The numerical value recognition unit 121 assigns a digit number assigned to an area including the center position of each partial area to each detected partial area.

Next, the numerical value recognition unit 121 determines whether or not each detected partial area can be used in the numerical value recognition process (step S205). The numerical value recognition unit 121 determines whether or not each partial area can be used in the numerical value recognition process based on whether or not each partial area includes blur or shine. A blur is a region where the difference in luminance value of each pixel in the image is small due to defocusing of the imaging device 104, or the same object is reflected in a plurality of pixels in the image due to a user's camera shake. It means an area where the difference in luminance value of each pixel is small. The term “shine” means a region where the luminance value of a pixel in a predetermined region in the image is saturated (out-of-white) due to the influence of disturbance light or the like.

Whether the numerical value recognition unit 121 includes blur in each partial region by a discriminator that is pre-learned to output a blur degree indicating the degree of blur included in the input image when the image is input. Determine whether or not. This discriminator is pre-learned using an image obtained by photographing a meter and not including blur by, for example, deep learning, and is stored in the storage device 110 in advance. Note that this discriminator may be pre-learned using an image obtained by photographing a meter and including blur. The numerical value recognition unit 121 inputs an image including the detected partial area to the classifier, and whether or not the partial area includes blur depending on whether or not the degree of blur output from the classifier is equal to or greater than the third threshold. Determine.

Alternatively, the numerical value recognition unit 121 may determine whether or not each partial area includes blur based on the edge strength of the luminance value of each pixel included in the partial area. The numerical value recognition unit 121 calculates the absolute value of the difference between the luminance values of the pixels adjacent to each other in the horizontal or vertical direction in the partial region or a plurality of pixels separated from the pixel by a predetermined distance as the edge strength of the pixel. . The numerical value recognition unit 121 determines whether or not the partial area includes blur depending on whether or not the average value of the edge intensities calculated for each pixel in the partial area is equal to or less than the fourth threshold value.

Alternatively, the numerical value recognition unit 121 may determine whether or not each partial area includes blur based on the distribution of luminance values of the pixels included in the partial area. The numerical value recognition unit 121 generates a histogram of the luminance value of each pixel in the partial region, and detects a local maximum value in each of the luminance value range indicating the numerical value (white) and the luminance value range indicating the background (black). Then, the average value of the full width at half maximum of each maximum value is calculated. The numerical value recognition unit 121 determines whether or not the partial area includes blur depending on whether or not the average value of the calculated half-value widths of the local maximum values is equal to or greater than the fifth threshold value.

In addition, the numerical value recognition unit 121 includes shine in each partial region by a discriminator that is pre-learned so as to output a degree of shine indicating the degree of shine included in the input image when an image is input. It is determined whether or not. This discriminator is pre-learned using, for example, an image obtained by photographing a meter and not including shine by deep learning or the like, and is stored in the storage device 110 in advance. Note that this discriminator may be pre-learned using an image obtained by photographing a meter and including shine. The numerical value recognition unit 121 inputs an image including the detected partial area to the classifier, and whether or not the partial area includes shine depending on whether or not the degree of shine output from the classifier is equal to or greater than a sixth threshold. Determine.

Alternatively, the numerical value recognition unit 121 may determine whether or not each partial area includes shine based on the luminance value of each pixel included in the partial area. The numerical value recognition unit 121 calculates the number of pixels whose luminance value is greater than or equal to the seventh threshold value (white) among the pixels in the partial region, and determines whether or not the calculated number is equal to or greater than the eighth threshold value. It is determined whether or not shine is included.

Alternatively, the numerical value recognition unit 121 may determine whether or not each partial area includes a shine based on the distribution of luminance values of each pixel included in the partial area. The numerical value recognition unit 121 generates a histogram of the luminance value of each pixel in the partial area, and determines whether or not the partial area depends on whether the number of pixels distributed in the area equal to or higher than the seventh threshold is equal to or higher than the eighth threshold. Whether or not is included is determined.

In addition, each threshold value and each range described above are set in advance by a prior experiment.

Next, the numerical value recognition unit 121 stores the information of the partial area determined to be usable in the management table of the storage device 110 (step S206), and ends a series of steps.

FIG. 6 is a diagram showing an example of the data structure of the management table.

In the management table, for each input image, information such as identification information (input image ID), evidence image information, partial area information, and digit value of each input image is stored in association with each other. The input image ID is uniquely assigned for each input image. The evidence image information is information indicating the storage destination of the evidence image. The evidence image is an image stored as an evidence image. As the evidence image, for example, an image obtained by cutting out an area within the numerical background frame from the input image is used. The evidence image is an example of at least a part of the input image. The partial area information is information indicating the storage destination of the partial area image, and is stored for each partial area corresponding to each digit. The partial area image is an image obtained by cutting out a partial area from the input image. The digit value is a digit value specified in each partial area, and is stored for each digit.

The numerical value recognition unit 121 cuts out each partial area determined to be usable from the input image, generates a partial area image, stores the partial area image in the storage device 110, and stores the storage destination in the management table as partial area information. . Note that the numerical value recognition unit 121 does not cut out each partial area determined to be usable from the input image, but may store position information of each partial area in the input image as partial area information in the management table. Good.

In addition, when there is a partial area determined to be usable in the input image, the numerical value recognition unit 121 generates an evidence image by cutting out the area in the numerical background frame from the input image, and stores it in the storage device 110. In addition to storing, the storage location is stored in the management table as evidence image information. Note that an image obtained by cutting out an area within the plate frame from the input image or the input image itself may be used as the evidence image. In that case, the numerical value recognition unit 121 stores the image obtained by cutting out the region in the plate frame from the input image or the input image itself as an evidence image in the storage device 110, and stores the storage location as evidence image information in the management table. To remember.

FIG. 7 is a flowchart showing an example of the operation of numerical value recognition processing. The operation flow shown in FIG. 7 is executed in step S104 of the flowchart shown in FIG. Each process of steps S301 to S309 in FIG. 7 is executed for each partial area determined to be usable in the numerical value recognition process. That is, each process of steps S301 to S309 in FIG. 7 is executed for each digit of the numerical value in the meter that is shown in the input image sequentially generated by the imaging device 104.

First, the numerical value recognition unit 121 determines whether or not the digit value of the digit number assigned to the partial area to be processed has been confirmed (step S301).

If the digit value of the digit number has been confirmed, the numerical value recognition unit 121 moves the process to step S310. On the other hand, when the digit value of the digit number has not been determined, the numerical value recognition unit 121 identifies the digit value reflected in the partial area (step S302).

The numerical value recognition unit 121, when an image including a single-digit numerical value is input, the digit value reflected in the partial region by a classifier that has been previously learned so as to output the numerical value reflected in the image. Is identified. This discriminator is pre-learned using a plurality of images obtained by photographing each numerical value in the meter, for example, by deep learning, and stored in the storage device 110 in advance. The numerical value recognition unit 121 inputs an image including the partial area to the discriminator, and specifies the numerical value output from the discriminator as a digit value reflected in the partial area. Note that the numerical value recognition unit 121 may specify a digit value shown in the partial area using a known OCR technique.

Next, the numerical value recognition unit 121 converts the specified numerical value into an integer (step S303). In general, a meter displays a measured value on a plate by rotating a cylindrical drum having a plurality of numbers printed on its side surface. Therefore, if the meter is photographed while the printed numbers are moving as the drum rotates, the partial area may include a plurality of numerical values. Therefore, the discriminator is pre-learned so as to output a decimal number according to the area in which each numerical value is shown when the partial area includes a plurality of numerical values. The numerical value recognition unit 121 rounds up, rounds, or rounds off the decimal part of the numerical value output from the discriminator, and converts the numerical value into an integer.

Next, the numerical value recognition unit 121 stores the specified numerical value in the management table of the storage device 110 as the digit value of the digit number assigned to the partial area (step S304).

Next, the numerical value recognition unit 121 determines whether or not numerical value recognition processing has been performed on a predetermined number (for example, 10) or more input images (step S305).

If the numerical value recognition process has not yet been executed for a predetermined number or more of input images, the numerical value recognition unit 121 moves the process to step S310. On the other hand, when the numerical value recognition processing is executed for a predetermined number or more of input images, the numerical value recognition unit 121 specifies the mode value among the digit values specified for the most recent predetermined number of input images ( Step S306). The numerical value recognizing unit 121 refers to the management table and identifies the most frequently stored digit value among the predetermined number of digit values closest to the digit number assigned to the partial area to be processed as the mode value. .

Next, the numerical value recognition unit 121 calculates the ratio of the number of occurrences (in the most recent predetermined number) of the specified mode value to the predetermined number (step S307).

Next, the numerical value recognition unit 121 determines whether or not the calculated ratio exceeds the ninth threshold value (step S308). For example, the ninth threshold is set to 50%.

When the calculated ratio does not exceed the ninth threshold value, the numerical value recognition unit 121 considers that the mode value is not yet a reliable value, and proceeds to step S310. On the other hand, when the calculated ratio exceeds the ninth threshold value, the numerical value recognition unit 121 identifies the identified mode value as a confirmed digit value (step S309). In this way, the numerical value recognition unit 121 determines the digit value only when the calculated ratio exceeds the ninth threshold value, so that the reliability of the recognized numerical value can be further increased.

Next, the numerical value recognition unit 121 determines whether or not the processing has been completed for all the detected partial areas (step S310).

If there is a partial area that has not been processed yet, the numerical value recognition unit 121 returns the process to step S301 and repeats the processes of steps S301 to S310. On the other hand, when the processing is completed for all the detected partial areas, the numerical value recognition unit 121 determines whether or not the digit values of all the digit numbers have been determined (step S311).

When the digit values of all the digit numbers have not yet been determined, the numerical value recognition unit 121 ends the series of steps without executing any particular process. On the other hand, when the digit values of all the digit numbers are confirmed, the numerical value recognition unit 121 recognizes the numerical value obtained by combining the confirmed digit values specified for each of all the digits as the numerical value in the meter (step S312). This step is finished.

As described above, the numerical value recognition unit 121 identifies and counts the numerical values in the meter shown in the sequentially generated input images for each digit, and recognizes the numerical values in the meter based on the counting results. The numerical value recognition unit 121 specifies the numerical values of other digits even for an input image in which a numerical value of a specific digit cannot be specified, and uses it for aggregation. Therefore, the numerical value in the meter can be accurately obtained using fewer input images. Can be recognized. Since the user does not need to continue to capture the meter until an input image that can identify all the digits is generated, the image processing apparatus 100 can improve user convenience.

Note that the numerical value recognition unit 121 may identify and count all the numerical values in the meter shown in each of the sequentially generated input images, and recognize the numerical values in the meter based on the counting result. .

In step S305, the numerical value recognizing unit 121 performs the processes in and after step S306 if the number of input images on which the numerical value recognition processing has been executed is not equal to or greater than a predetermined number, but the number of digits can be determined. May be executed. For example, when the predetermined number is 10 and the ninth threshold value is 50%, the digit values specified for each input image are all the same when the number of input images subjected to the numerical value recognition process is six. For example, the digit value is the mode value, and the ratio of the number of occurrences of the mode value is 60% or more. In such a case, the numerical value recognition unit 121 may determine the numerical value to be recognized even if the number of input images on which numerical value recognition processing has been executed is not equal to or greater than a predetermined number. Thereby, the numerical value recognition part 121 can shorten the recognition time by numerical value recognition processing.

FIG. 8 is a flowchart showing an example of operation of evidence image storage processing. The operation flow shown in FIG. 8 is executed in step S106 of the flowchart shown in FIG.

First, the determination unit 122 determines whether each partial region is clear for every partial region detected by the numerical value recognition unit 121 (step S401). Note that the determination unit 122 determines whether or not each partial region is clear only for the partial region in which the digit value specified by the numerical value recognition unit 121 matches the corresponding digit value in the numerical value recognized by the numerical value recognition unit 121. You may judge.

The clear partial area means that the numerical value included in the partial area can be recognized, and that the partial area does not include blur or shine. On the contrary, that the partial area is unclear means that the numerical value included in the partial area cannot be recognized, and means that the partial area includes blur or shine. The determination unit 122 determines whether each partial area is clear depending on whether each partial area includes blur or shine.

In the same way as the process of step S205, the determination unit 122 determines whether each partial region includes blur or shine by using a discriminator that has captured a meter and previously learned using at least an image that does not include blur or shine. Determine whether. Alternatively, the determination unit 122 determines whether each partial area includes blur or shine, based on the luminance value of the image in each partial area, in the same manner as the process of step S205. However, the determination unit 122 makes it easier to determine that each partial area includes blur or shine than in the process of step S205. That is, the determination unit 122 makes the third threshold value, the fifth threshold value, the sixth threshold value, or the eighth threshold value smaller than the value used in the process of step S205, and the fourth threshold value is a value used in the process of step S205. Larger than.

Note that the determination unit 122 may determine whether or not each partial region includes blur or shine according to the same determination criterion as that in step S205. In that case, the determination unit 122 may determine whether each partial region includes blur or shine using the determination result in the process of step S205. Thereby, the determination part 122 can shorten the processing time by determination processing, and can reduce processing load.

Next, the control unit 123 determines whether or not there is one input image for which it is determined that the partial areas corresponding to all the digits are clear (step S402).

When there is one input image in which the partial area corresponding to all the digits is determined to be clear, the control unit 123 selects the one input image. The control unit 123 stores only the evidence image corresponding to the selected one input image as an evidence image, associates the numerical values recognized by the numerical value recognition unit 121 with each other, and stores them in the storage device 110 (step S403). End the step.

On the other hand, when there is no input image in which the partial area corresponding to all the digits is determined to be clear, the control unit 123 determines whether a partial evidence image is used as the evidence image. A digit whose region is not determined to be clear is identified (step S404). When the process of step S404 is first executed, all the digits are specified as digits for which the partial area is not determined to be clear.

Next, the control unit 123 selects an input image including the largest number of partial areas determined to be clear for the specified digit as an input image using the corresponding evidence image as the evidence image (step S405). ). Thus, the control unit 123 selects an input image for which each partial area is determined to have a clear partial area corresponding to each digit.

Next, the control unit 123 determines whether or not there remains a digit for which the partial area is not determined to be clear in any of the selected input images (step S406).

If there are still digits that have not been determined to be clear, the control unit 123 returns the process to step S404 to re-specify digits that have not been determined to be clear. On the other hand, if there are no remaining digits that are not determined to be clear in the partial area, the control unit 123 stores the evidence images corresponding to the plurality of input images selected in step S405 in the storage device 110 as evidence images. Store (step S407). And the control part 123 complete | finishes a series of steps. The control unit 123 stores each evidence image in the storage device 110 in association with the numerical value recognized by the numerical value recognition unit 121.

FIG. 9 is a diagram for explaining a combination of evidence images stored in the storage device 110.

9 shows a combination when it is determined that the partial areas corresponding to all the digits (digits 1 to 4) in the input image 1 are clear. In this case, the evidence image of the input image 1 is used as the evidence image. The combination 2 corresponds to 3 digits (digits 1 to 3) in the input image 1 where there is no input image in which the partial areas corresponding to all the digits (digits 1 to 4) are clear. A combination when it is determined that only the partial area is clear is shown. In this case, the evidence image of the input image 1 and the evidence image of the input image 2 in which the partial area corresponding to the digit 4 that is not determined to be clear in the input image 1 is clear are evidence. Used as an image.

In the combinations 3 and 4, there is no input image in which the partial area corresponding to 3 digits is determined to be clear, and only the partial area corresponding to 2 digits (digit 1 and digit 2) in the input image 1 is clear. A combination in a case where it is determined that there is one is shown. A combination 3 indicates a combination in a case where the partial areas corresponding to the digits 3 and 4 that are not determined to be clear in the input image 1 are determined to be clear in the input image 2. In this case, the evidence image of the input image 1 and the evidence image of the input image 2 are used as evidence images. On the other hand, the combination 4 indicates a combination in the case where there is no input image in which the partial areas corresponding to the digits 3 and 4 that are not determined to be clear in the input image 1 are determined to be clear. In this case, it is determined that the evidence image of the input image 1, the evidence image of the input image 2 determined to have a clear partial area corresponding to the digit 3, and the partial area corresponding to the digit 4 are clear. The evidence image of the input image 3 is used as the evidence image.

Combination 5 indicates a combination when there is no input image determined that the partial area corresponding to 2 digits is clear. In this case, the evidence images of the input images 1 to 4 that are determined to have clear partial areas corresponding to the digits 1 to 4 are used as evidence images.

As described above, the control unit 123 uses, as evidence images, the evidence images corresponding to any of the input images determined to be clear for each of the partial regions corresponding to all the digits. use. In particular, the control unit 123 uses, as an evidence image, an evidence image of another input image that complements a partial region that has not been determined to be clear in an input image in which the evidence image is used as an evidence image. As a result, the control unit 123 stores a plurality of images having evidence ability even when there is no input image in which all the partial areas are determined to be clear, thereby enabling an appropriate image as an evidence image. Can be stored efficiently. The user does not need to continue photographing the meter until an input image determined that all the partial areas are clear is generated, and the image processing apparatus 100 can improve user convenience. . In addition, the image processing apparatus 100 does not need to store images related to all input images used in the numerical value recognition process, and can reduce the storage capacity of the storage device 110.

In addition, when the criterion for determining whether or not the partial area can be used in the numerical value recognition process is different from the criterion for determining whether the partial region is clear, even after the numerical value recognition unit 121 recognizes the numerical value in the meter, There is a possibility that evidence images in which the partial areas corresponding to the digits are clear are not prepared. In that case, after the numerical value recognition unit 121 recognizes the numerical value in the meter, the control unit 123 causes the imaging device 104 to image the meter until evidence images in which the partial areas corresponding to all the digits are clear are prepared. The input image may be continuously generated. Since the numerical value recognition unit 121 has already recognized the numerical values in the meter, the numerical value recognition unit 121 stops counting the numerical values in the meter. However, the control unit 123 determines whether each partial area is clear for the newly generated input image. Continue the determination. When the evidence images in which the partial areas corresponding to all the digits are clear are prepared, the control unit 123 causes the imaging device 104 to stop the imaging of the meter and ends the evidence image storage process. As a result, the image processing apparatus 100 can acquire an evidence image that can be visually identified more reliably.

In addition, the control unit 123, when the numerical image recognition unit 121 recognizes the numerical value in the meter, when the evidence image in which the partial areas corresponding to all the digits are clear is not prepared within a predetermined time, the meter by the user A shooting stop instruction for instructing to stop shooting may be received. In that case, the control unit 123 displays display data such as a button for accepting a photographing stop instruction from the user on the display device 103. When the user taps the button using the input device 102 to input a shooting stop instruction and receives a shooting stop instruction signal from the input device 102, the control unit 123 receives the shooting stop instruction.

When receiving a shooting stop instruction, the control unit 123 selects, for each digit, a partial region determined to be the clearest from the partial regions corresponding to each digit, and includes each selected partial region. The evidence image of each input image is used as the evidence image. The control unit 123, for example, has the clearest partial area with the smallest degree of blur output from each classifier, the partial area with the largest average value of edge strength, or the partial area with the smallest average value of half width. Select as a partial area. Alternatively, the control unit 123 outputs pixels that are distributed in the partial region with the smallest degree of shine output from each classifier, the number of pixels having a luminance value equal to or higher than the seventh threshold value, or the region equal to or higher than the seventh threshold value in the histogram. The partial area having the smallest number is selected as the partial area that is the clearest. When a clear evidence image is not acquired within a predetermined time, the user can confirm the clearest image among the images acquired so far, and use the image as the evidence image as it is. It is possible to determine whether to re-shoot the meter.

As described above in detail, the image processing apparatus 100 uses, for each digit of the numerical value in the meter, an evidence image corresponding to the input image determined to have a clear partial area corresponding to each digit as an evidence image. Remember. As a result, the image processing apparatus 100 can efficiently store an appropriate image as an evidence image.

FIG. 10 is a block diagram showing a schematic configuration of the processing circuit 230 in the image processing apparatus according to another embodiment.

The processing circuit 230 is used instead of the processing circuit 130 of the image processing apparatus 100, and executes the entire processing instead of the CPU 120. The processing circuit 230 includes a numerical value recognition circuit 231, a determination circuit 232, a control circuit 233, and the like.

The numerical value recognition circuit 231 is an example of a numerical value recognition unit, and has the same function as the numerical value recognition unit 121. The numerical value recognition circuit 231 sequentially acquires input images obtained by photographing the meter from the imaging device 104 and sequentially stores them in the storage device 110. In addition, the numerical value recognition circuit 231 identifies and counts the numerical values in the meter shown in each input image, recognizes the numerical values in the meter based on the totaled results, and stores the recognition results in the storage device 110.

The determination circuit 232 is an example of a determination unit and has the same function as the determination unit 122. The determination circuit 232 determines whether or not each partial area is clear for each partial area corresponding to each digit of the numerical value in the meter, and outputs the determination result to the control circuit 233.

The control circuit 233 is an example of a control unit and has the same function as the control unit 123. For each digit, the control circuit 233 selects an input image for which the partial area corresponding to each digit is determined to be clear, and uses the evidence image corresponding to the selected input image as the evidence image for the numerical value recognition circuit 231. Are stored in the storage device 110 in association with each other.

As described in detail above, the image processing apparatus 100 can efficiently store an appropriate image as an evidence image even when the processing circuit 230 is used.

The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, each discriminator used in the partial area detection process, the numerical value recognition process, or the evidence image storage process may be stored in an external device such as a server device instead of being stored in the storage device 110. In that case, the numerical value recognition unit 121 transmits each image to the server apparatus via the communication apparatus 101, and receives and acquires the identification result output from each classifier from the server apparatus.

Further, the image processing apparatus 100 is not limited to a portable information processing apparatus, and may be, for example, a fixed point camera or the like installed so that a meter can be imaged.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 104 Imaging apparatus 110 Storage apparatus 121 Numerical value recognition part 122 Judgment part 123 Control part

Claims

A storage unit;
An imaging unit that sequentially generates input images obtained by imaging the meter;
A numerical value recognizing unit that identifies and counts the numerical values in the meter shown in each of the sequentially generated input images, and recognizes the numerical values in the meter based on the totaling results;
A determination unit that determines whether each partial area is clear for each partial area corresponding to each digit of the numerical value in the meter;
For each digit, select an input image in which the partial area corresponding to each digit is determined to be clear, and associate the numerical value recognized by the numerical value recognition unit with at least a part of the selected input image as an evidence image A control unit for storing in the storage unit
An image processing apparatus comprising:
When there is one input image in which all the partial areas are determined to be clear, the control unit stores at least a part of only the one input image as an evidence image in the storage unit. 2. The image according to claim 1, wherein when there is no input image determined to be clear in all partial areas, at least a part of each of the plurality of input images is stored in the storage unit as an evidence image. Processing equipment.
3. The image processing apparatus according to claim 1, wherein the determination unit determines whether each partial region is clear or not by a discriminator preliminarily learned using at least a clear image taken by the meter. .
3. The image processing apparatus according to claim 1, wherein the determination unit determines whether each partial area is clear based on a luminance value of an image in each partial area.
The numerical value recognition unit
For each digit of the numerical value in the meter shown in each sequentially generated input image, specify a digit value, specify the mode value among the specified digit values for a predetermined number of input images, If the ratio of the number of occurrences of the specified mode value to the predetermined number exceeds a threshold value, the specified mode value is specified as a fixed digit value,
The image processing apparatus according to any one of claims 1 to 4, wherein a numerical value obtained by combining fixed digit values specified for all of the digits is recognized as a numerical value in the meter.
A control method for an image processing apparatus, comprising: a storage unit; and an imaging unit that sequentially generates input images obtained by capturing a meter,
Identify and count the numerical values in the meter that appear in each of the sequentially generated input images, and based on the totaling results, recognize the numerical values in the meter,
For each partial area corresponding to each digit of the numerical value in the meter, determine whether each partial area is clear,
For each digit, select an input image in which a partial area corresponding to each digit is determined to be clear, associate at least a part of the selected input image as an evidence image with the recognized numerical value, and store the storage unit Remember
A control method comprising:
A control program for an image processing apparatus, comprising: a storage unit; and an imaging unit that sequentially generates input images obtained by photographing a meter,
Identify and count the numerical values in the meter that appear in each of the sequentially generated input images, and based on the totaling results, recognize the numerical values in the meter,
For each partial area corresponding to each digit of the numerical value in the meter, determine whether each partial area is clear,
For each digit, select an input image in which a partial area corresponding to each digit is determined to be clear, associate at least a part of the selected input image as an evidence image with the recognized numerical value, and store the storage unit Remember
A control program that causes the image processing apparatus to execute