JP7279397B2 - Image processing device, image processing program, image processing method, and information collection system - Google Patents

Description

The present invention relates to an image processing device, an image processing program, an image processing method, and an information collection system.

For example, an analog meter is an instrument with a dial and a pointer that moves along the scale of the dial to measure a target measurement. Such analog meters are used in various situations. For example, they are installed in equipment in facilities such as factories, and workers monitor the equipment while visually checking the measured values of the analog meter. It is often used in such cases.

2. Description of the Related Art In recent years, facilities such as factories are required to be able to automatically read the measured values of analog meters and to collect the data as one way of reducing operating costs. There are various methods for automatically reading analog meters, and examples thereof include, for example, replacing analog meters and automating the work of visually recognizing analog meters.

Replacing an analog meter is a method of replacing the currently used analog meter with a dedicated meter that outputs the indicated value indicated by the pointer as digital data. In this case, it is necessary to replace the currently used analog meter with a dedicated meter, and there is a problem that equipment must be stopped during the replacement period.

Automating the visual recognition work of an analog meter involves photographing the pointer and dial of the analog meter with a camera, processing the photographed image, and reading the indicated value indicated by the pointer. In this case, reading of the indicated value can be automated by installing an indicated value reading device including a camera that captures the analog meter and an image processing device that processes the captured image in the facility. By introducing such an indicated value reading device, there is an advantage that it is not necessary to stop equipment, for example, so there is a strong demand.

Conventionally, there is a technique disclosed in Patent Document 1, for example, which relates to the automation of visual recognition work for an analog meter. In the technology described in Patent Document 1, in order to detect the position of a pointer appearing in an image captured by a camera, the entire captured image is binarized and a pixel indicating black in the image is searched. Then, it is determined that the pointer is positioned in the area where the largest number of black pixels are present.

By the way, it is necessary to install the indicated value reading device described above in each analog meter. Therefore, in a facility such as a factory where there are a large number of analog meters, the introduction cost of the meter reading device becomes high, and it is demanded to keep the introduction cost low.

JP 2017-126187 A

However, the technique described in Patent Literature 1 described above requires binarization processing of the entire captured image and pixel search in the entire image, so the amount of arithmetic processing becomes enormous and the load related to image processing is large. There is a problem. In addition, it is necessary to use high-performance and expensive arithmetic processing units and cameras, and there is also the problem that it is difficult to reduce the introduction cost.

Therefore, in view of the above problems, the present invention provides an image processing apparatus, an image processing program, and an image processing that can read a value indicated by an indicator needle of an analog meter while suppressing the amount of calculation and memory usage required for image processing. It is intended to provide a method and information gathering system.

In order to solve this problem, an image processing apparatus according to a first aspect of the present invention provides a posture information detection jig having a plurality of vertices and a vertex distinguishable from other vertices among the plurality of vertices as a feature point. (1) A virtual reference image of the pointer instrument and an input captured image Using the parameters of projective transformation with the image of the pointer instrument, the pointer detection coordinate string having a plurality of coordinate values indicating the position of the pointer of the pointer instrument in the reference image, or the projective transformation of the captured image and (2) a plurality of pixel values extracted from the captured image using the pointer detection coordinate sequence that has been projectively transformed, or a plurality of pixels extracted from the captured image that has been projectively transformed using the pointer detection coordinate sequence. (3) an indicator value derivation unit for deriving an indicator value corresponding to the position of the indicator needle detected by the indicator needle detection unit; (4) an input an orientation information detection unit that detects each coordinate value of a plurality of vertices of the orientation information detection jig shown in the captured image from the captured image, and specifies the orientation of the pointer instrument based on the position of the feature point; 5) a projective transformation parameter deriving unit for deriving a projective transformation parameter based on each coordinate value of a plurality of reference points forming a reference image and each coordinate value of each of a plurality of vertices detected by the posture information detecting unit; It is characterized by having

An image processing program according to a second aspect of the present invention is an indicator needle instrument having a plurality of vertices and having , as a feature point, a vertex that is distinguishable from other vertices among the plurality of vertices. In the image processing program for deriving the indicated value of the pointer gauge from the captured image of the display surface, the computer is provided with (1) a virtual reference image of the pointer gauge and the pointer gauge in the input captured image. A projective transformation unit that projectively transforms a pointer detection coordinate string having a plurality of coordinate values indicating the position of the pointer of the pointer instrument in the reference image or the captured image using the projective transformation parameter with the image; ) From a plurality of pixel values extracted from the captured image using the pointer detection coordinate sequence that has been projectively transformed, or from a plurality of pixel values extracted from the captured image that has been projectively transformed using the pointer detection coordinate sequence, the pointer (3) an indicator value derivation unit for deriving an indicator value corresponding to the position of the pointer detected by the pointer detection unit; (4) from an input captured image; (5) a reference image ; and each coordinate value of a plurality of vertices detected by the orientation information detection unit. and

An image processing method according to a third aspect of the present invention is directed to an indicator needle instrument having a plurality of vertices and having , as a feature point, a vertex distinguishable from other vertices among the plurality of vertices, to which a posture information detection jig is attached. In an image processing method for deriving an indicated value of a pointer gauge from a captured image of a display surface, (1) a projective transformation unit converts a virtual reference image of the pointer gauge and a pointer in the input captured image into projectively transforming the pointer detection coordinate string having a plurality of coordinate values indicating the position of the pointer of the pointer instrument in the reference image or the captured image using the projective transformation parameter with the image of the instrument, and (2) indicating A plurality of pixel values extracted from the captured image by the pointer detection unit using the pointer detection coordinate string that has undergone projective transformation, or a plurality of pixel values that have been extracted from the captured image that has undergone projective transformation using the pointer detection coordinate sequence. (3) an indication value derivation unit derives an indication value corresponding to the position of the pointer detected by the pointer detection unit; (4) an attitude information detection unit inputs (5 ) projection The transformation parameter derivation unit derives the projective transformation parameter based on each coordinate value of a plurality of reference points forming the reference image and each coordinate value of each of a plurality of vertices detected by the posture information detection unit. and

An information gathering system according to a fourth aspect of the present invention comprises: (1) an image processing device for deriving an indicated value of the pointer instrument from a captured image of the display surface of the pointer instrument; and (2) derivation by the image processing device. (3) a second communication device for receiving the signal transmitted from the first communication device; (4) a second and a management device for storing the instruction value included in the signal received by the communication device, wherein the image processing device is the image processing device according to the first aspect of the present invention.

According to the present invention, it is possible to read the value indicated by the pointer of the analog meter while suppressing the amount of calculation and memory usage required for image processing.

1 is a configuration diagram showing the overall configuration of an information collection system according to a first embodiment; FIG. FIG. 4 is an explanatory diagram illustrating an example of a relative positional relationship between a camera that captures an image of an analog meter and the analog meter and an example of a captured image (No. 1); 2 is an internal configuration diagram showing the internal configuration of the image processing apparatus according to the first embodiment; FIG. FIG. 4 is an explanatory diagram for explaining posture reference information according to the first embodiment; FIG. 4 is an explanatory diagram for explaining pointer detection coordinate sequences according to the first embodiment; 4 is a flowchart showing operations of an image processing method according to the first embodiment; 4 is a flowchart showing a method of detecting four vertex coordinates by an orientation information detection unit according to the first embodiment; FIG. 4 is an explanatory diagram illustrating a method of detecting four vertex coordinates by an orientation information detection unit according to the first embodiment; FIG. 4 is an explanatory diagram for explaining the relationship between pointer detection pixel value sequences and pointer detection coordinate sequences in the first embodiment; FIG. 4 is an explanatory diagram illustrating an example of a relative positional relationship between a camera that captures an image of an analog meter and the analog meter and an example of a captured image (No. 2); FIG. 7 is an internal configuration diagram showing the internal configuration of an image processing apparatus according to the second embodiment; FIG. 11 is a configuration diagram showing the configuration of an orientation information detection jig according to the second embodiment; FIG. 11 is an explanatory diagram illustrating a state in which the posture information detection jig according to the second embodiment is attached to an analog meter; 9 is a flowchart showing image processing operations performed by the image processing apparatus according to the second embodiment; FIG. 10 is an explanatory diagram for explaining projective transformation in which four reference points of posture reference information are used as transformation sources and four vertices of a posture information detection jig are used as transformation destinations; 9 is a flowchart showing a detection method of an orientation information detection jig of an orientation information detection unit according to the second embodiment; FIG. 11 is an explanatory diagram illustrating a method of labeling four vertices according to the second embodiment; FIG. 11 is a configuration diagram showing the configuration of an orientation information detection jig according to a third embodiment; FIG. 11 is an explanatory diagram illustrating a state in which the posture information detection jig according to the third embodiment is attached to an analog meter; 14 is a flowchart showing a detection method of an orientation information detection jig of an orientation information detection unit according to the third embodiment; FIG. 11 is an internal configuration diagram showing the internal configuration of an image processing apparatus according to a fourth embodiment; 10 is a flow chart showing image processing in an image processing apparatus according to a fourth embodiment; FIG. 4 is a configuration diagram illustrating the configuration of a table showing the correspondence relationship between detected pointer angles and indicated values in the first embodiment; FIG. 11 is an explanatory diagram illustrating a case of determining pixel values of a fixed area including each coordinate value of a pointer detection coordinate sequence according to a modified embodiment; FIG. 11 is a configuration diagram showing the configuration of a dial and an indicator needle of an analog meter and the configuration of posture reference information according to a modified embodiment;

(A) First Embodiment Hereinafter, a first embodiment of an image processing apparatus, an image processing program, an image processing method, and an information collection system according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of First Embodiment (A-1-1) Overall Configuration of Information Collection System FIG. 1 is a configuration diagram showing the overall configuration of an information collection system according to the first embodiment.

In FIG. 1 , an information collection system 10 according to the first embodiment has an analog meter 1 , a camera 2 , an image processing device 3 , communication devices 4 and 5 and a management device 6 .

The information collection system 10 captures the analog meter 1 with the camera 2, acquires the indication value indicated by the pointer of the analog meter 1 from the captured image, reads the indication value, and collects the indication value. is. As a result, the indicated value of the analog meter 1 can be read in real time, and the indicated value can be accumulated and held as history information.

The analog meter 1 is an indicator-needle instrument that measures a measured value by moving a dial and an indicator along the scale arranged on the dial. For example, the analog meter 1 may be installed in equipment or the like in a facility such as a factory, and may be a meter such as an analog voltmeter, analog ammeter, or the like. In the analog meter 1, the pointer rotates in the scale area of the dial about the rotation axis, or the dial has a vertical or horizontal direction and the pointer moves in the vertical or horizontal direction. However, in this embodiment, a type in which the pointer rotates in the scale area of the dial will be described as an example.

The camera 2 is an imaging device for imaging the analog meter 1 and provides a captured image to the image processing device 3 connected thereto. For the camera 2, for example, a general-purpose camera that does not have a function of correcting captured images can be applied. As a result, since an inexpensive camera can be used, the introduction cost of the indicated value reading system can be suppressed.

The image processing device 3 acquires a captured image including the analog meter 1 from the camera 2, analyzes the captured image, and derives an indicated value of the pointer in the captured image. Further, the image processing device 3 provides the communication device 4 with data including the instruction value. Although the detailed description of the image processing device 3 will be given later, according to this embodiment, the processing load of the instruction value calculation processing can be suppressed, so that a general-purpose arithmetic processing device such as a microcomputer can be used. The introduction cost of the indicated value reading system can be suppressed.

The communication devices 4 and 5 are communication devices that transmit and receive signals including indicated value data of the analog meter 1 . The communication device 4 transmits a communication signal including the indicated value data of the analog meter 1 acquired from the image processing device 3. The communication device 5 receives the communication signal transmitted from the communication device 4, and performs the communication. It gives the indication value data contained in the signal to the management device 6 . The communication devices 4 and 5 can be those that perform wireless communication according to, for example, a wireless communication protocol for multi-hop wireless communication. Note that the wireless communication protocol is not particularly limited. Furthermore, the communication devices 4 and 5 may communicate a signal including the indicated value data of the analog meter 1 by wired communication.

The management device 6 records the indicated value data of the analog meter 1 given from the communication device 5 and manages the indicated value data of the analog meter 1 . The management device 6 has an arithmetic processing unit such as a CPU, display means such as a monitor, etc., and displays the indicated value data of the analog meter 1 in real time, and performs statistical processing using historical indicated value data. etc., and display the data. The management method of the management device 6 using the indicated value data is not particularly limited, and various methods can be applied.

(A-1-2) Example of Image Captured by Camera FIG. 2 is an explanatory diagram illustrating an example of the relative positional relationship between the camera 2 that captures the analog meter 1 and the analog meter 1 and an example of the captured image.

Since the analog meter 1 is often installed, for example, in equipment or the like, it is assumed that the position and orientation of the analog meter 1 cannot be changed. When analyzing the captured image of the camera 2 and calculating the indication value of the pointer of the analog meter 1, it is desirable that the camera 2 is arranged in front of the analog meter 1, but such a positional relationship is not necessarily the case. There is also

In this embodiment, since it is assumed that a general-purpose camera 2 is used, the captured image is not corrected according to the attitude of the camera 2 or the relative positional relationship with the analog meter 1 .

Therefore, first, the relative positional relationship between the analog meter 1 and the camera 2 will be described, and the image of the analog meter 1 displayed in the captured image in such a positional relationship will be described.

FIG. 2A illustrates a case where the camera 2 is arranged in front of the analog meter 1. FIG. In this case, the camera 2 can photograph the panel portion of the analog meter 1 (the portion where the dial and pointer are located) from the front, and a captured image P1 as shown in FIG. 2(B) is obtained. At this time, the image of the analog meter 1 displayed in the captured image P1 is in an upright posture.

The image captured by the camera 2 has a certain size in the image coordinate system, and can be represented by coordinate values (XY coordinate values) on the XY plane of the image coordinate system.

On the other hand, FIG. 2C illustrates a case where the camera 2 is arranged obliquely to the left with respect to the analog meter 1 . In this case, the camera 2 takes an image of the panel portion of the analog meter 1 from an oblique left direction, so a captured image P2 as shown in FIG. 2(D) is obtained. That is, the image of the analog meter 1 displayed in the captured image P2 assumes an obliquely deformed posture.

In this way, the position of the camera 2 and the attitude of the camera 2 with respect to the analog meter 1 can change the attitude of the analog meter 1 displayed in the captured image. When calculating, it is necessary to consider the posture and deformation of the analog meter 1 in the captured image.

(A-1-3) Detailed Configuration of Image Processing Apparatus FIG. 3 is an internal configuration diagram showing the internal configuration of the image processing apparatus 3 according to the first embodiment.

3, the image processing apparatus 3 according to the first embodiment includes an image input unit 31, a posture information detection unit 32, a storage unit 33, a projective transformation parameter calculation unit 34, a needle detection coordinate sequence projective transformation unit 35, an instruction It has a needle detection section 36 , an indicated value calculation section 37 and an output section 38 .

The hardware of the image processing device 3 includes, for example, a CPU, a ROM, a RAM, an EEPROM, an input/output interface, etc. The CPU executes a processing program (eg, an image processing program) stored in the ROM. As a result, various functions are realized. It should be noted that the image processing program may be installed in a computer such as a CPU to construct the image processing.

The image input unit 31 inputs a captured image including the analog meter 1 from the camera 2 that captures the analog meter 1 .

The orientation information detection unit 32 detects the coordinate values of the vertices of the analog meter 1 in the captured image (actual image) in order to detect the orientation of the analog meter 1 shown in the input captured image. is.

Here, the coordinate values of the vertices detected by the posture information detection unit 32 are also called "posture information". In the first embodiment, the posture information detection unit 32 detects the rectangular surface shape including the dial and pointer in the outer frame of the box-shaped analog meter 1, and detects the four vertices of the surface. Detect coordinates.

For example, when the camera 2 is provided in front of the box-shaped analog meter 1, the camera 2 can photograph the analog meter 1 from the front. In that case, the outer frame of the substantially rectangular analog meter 1 is reflected in the captured image. , and the coordinate values of each vertex may be detected. Further, for example, when the camera 2 photographs the analog meter 1 from a direction oblique to the analog meter 1, the image of the analog meter 1 is reflected in a deformed state such as a trapezoid. In this case, the posture information detection unit 32 determines the four sides forming the trapezoid, sets the intersection of each side as a vertex, and detects the coordinate values of each vertex. In any case, the orientation information detection unit 32 identifies the position and orientation of the analog meter 1 by identifying the outer frame of the analog meter 1 reflected in the captured image, and calculates the coordinate values of the four vertices. detect it.

The storage unit 33 stores preset posture reference information 331, pointer detection coordinate sequence 332, and the like.

The attitude reference information 331 is information including coordinate values of a plurality of reference points used for projective transformation processing. For example, it is information about coordinate values indicating the positions of four reference points in the reference image. The pointer detection coordinate column 332 is information for specifying the position of the pointer of the analog meter 1 in the reference image. The detailed configurations of the posture reference information 331 and pointer detection coordinate sequence 332 will be described later.

Here, the reference image includes a virtual image in which the analog meter 1 is ideally upright. The posture of the analog meter 1 reflected in the captured image of the analog meter 1 captured by the camera 2 can be distorted or deformed depending on the relative positional relationship with the camera 2, the posture of the camera 2, or the like. The image is a virtual image in which the analog meter 1 is ideally upright and is not distorted or rotated. Any "reference image" described in the following description is virtual.

In this way, the posture reference information 331 stored in the storage unit 33 is four reference points in the image of the analog meter 1 that is upright and not distorted, rotated, etc., reflected in the virtual reference image. This is information about coordinate values indicating a position. In other words, in the first embodiment, the image processing apparatus 3 may store the orientation reference information 331, which is information about coordinate values, in the storage unit 33. For example, image information corresponding to the reference image (brightness and color information) (including information that enables the image to be drawn) need not be stored.

The projective transformation parameter calculation unit 34 converts the four coordinates detected by the posture information detection unit 32 (that is, the four coordinates in the actual image captured by the camera 2) and preset posture reference information 331 into Based on this, projective transformation parameters (also called projective transformation coefficients) are derived.

For example, the projective transformation parameter converts a virtual reference image (for example, a two-dimensional plane) in which the analog meter 1 ideally standing upright is captured by the camera 2 into the image area of the analog meter 1 ( For example, it is a parameter used when projecting onto a two-dimensional plane), and generally consists of eight unknowns. In order to determine these eight coefficients, the projective transformation parameter calculation unit 34 calculates the values of the four vertex coordinates detected by the posture information detection unit 32 and the coordinate values of the four reference values of the posture reference information 331. Calculated using

The needle detection coordinate sequence projective transformation unit 35 uses the projective transformation parameters derived by the projective transformation parameter calculation unit 34 to projectively transform all the coordinate values of the pointer detection coordinate sequence 332 in the reference image. . As a result, all the coordinate values of the pointer detection coordinate sequence in the reference image are projectively transformed from the reference image to the actual image.

The pointer detection unit 36 acquires pixel values from the captured image (original image) from the camera 2 based on each coordinate value of the pointer detection coordinate string after projective transformation, and consists of all the acquired pixel values. Create a needle detection pixel value sequence to be used. Then, the pointer detection unit 36 detects the position of the pointer from among all the pixel values forming the pointer detection pixel value sequence. A detailed description of the pointer detector 36 will be given later.

The indicator value calculation unit 37 refers to the pointer detection coordinate string 332, calculates the pointer angle corresponding to the coordinates detected by the pointer detection unit 36, and outputs an instruction value based on the pointer angle. is.

(A-1-4) Attitude Reference Information 331 and Needle Detection Coordinate Array 332
FIG. 4 is an explanatory diagram for explaining the attitude reference information 331 according to the first embodiment.

FIG. 4A exemplifies the position coordinates of four reference points forming the orientation reference information 331 in the reference image.

As shown in FIG. 4A, indices are assigned to the four reference points, for example, point Ar, point Br, point Cr, and point Dr. This index assignment may be arbitrarily assigned. Then, as illustrated in FIG. 4B, coordinate values (0, 0), (0, 400), (400, 400), ( 400, 0) are stored in the storage unit 33 in association with the index.

Note that the coordinate system of the reference image may be the same as the coordinate system of the captured image captured by the camera 2 . The four reference points that constitute the posture reference coordinates can be preset arbitrarily selected points in the coordinate system of the image captured by the camera 2 . In the example of FIG. 4A, a square is formed in the reference image, and four vertices of the square are used as reference points. The size of the square in the reference image is not particularly limited.

FIG. 5 is an explanatory diagram for explaining the needle detection coordinate sequence 332 according to the first embodiment.

As shown in FIG. 5B, the needle detection coordinate column 332 is information having items such as "index", "coordinate", and "pointer angle".

The pointer detection coordinate column 332 gives an index (label) to each position coordinate where the pointer can exist within the existing area of the rotating pointer in order to specify the position value of the pointer in the reference image. . Each index, the XY coordinate value of each index, and the angle of the pointer of the analog meter 1 are associated with each other and created. In other words, the needle detection coordinate string 332 has a structure in which a plurality of data are connected, with each unit of data pairing the XY coordinate value of each pointer detection coordinate (index) and the pointer angle value at that time. It's becoming

For example, it is assumed that the analog meter 1 has a dial and pointer as shown in FIG. 5(C). In the reference image, an index "G1" is given to the position corresponding to the position of the scale "0" of the analog meter 1 in FIG. 5(C). In this case, the XY coordinate values of index "G1" are (100, 100). Furthermore, when the position of the pointer is at "G1", it corresponds to the scale "0" in the analog meter 1, so "pointer angle: 0" is set.

After that, indexes G2, G3, . , and the pointer angle is associated with each index. Then, as shown in FIG. 5A, each index forming pointer detection coordinate row 332 is formed on virtual circle VC2. Note that the size of the unit angle described above may be arbitrary.

Here, the reason why the virtual circle VC2 forming the pointer detection coordinate sequence 332 is formed within the presence area of the pointer position will be described. For example, in the image captured by the camera 2, the pointer may not cross the scale of the analog meter 1, and in such a case, the indicated value indicated by the pointer may not be read. However, as in the first embodiment, by forming the virtual circle VC2 that constitutes the needle detection coordinate sequence in the existing region of the pointer position, that is, the radius of the virtual circle VC2 is larger than that of the virtual circle VC1. By decreasing the distance between the pointer and the scale, the position coordinates of the pointer in the image can be detected and the angle of the pointer can be detected even when the pointer and the scale do not intersect.

Since the virtual circle VC2 that forms the needle detection coordinate sequence 332 is formed in the existence area of the position of the pointer, it is formed within the area of the virtual circle VC1 that is formed at the position of each scale of the analog meter 1. will be made. In other words, the radius of the virtual circle VC2 with the point S as its center is smaller than the radius of the virtual circle VC1 with the point S as its center.

A point S is the center of a square formed by four reference points that constitute the posture reference coordinates. It is the intersection with the minute. A point S is a point corresponding to the center of rotation of the pointer in the analog meter 1 . Therefore, the point S has a positional relationship such that the center of the square formed by the four reference points of the posture reference information 331 and the center of the virtual circle VC2 forming the needle detection coordinate sequence 332 are aligned.

(A-2) Operation of First Embodiment Next, the operation of image processing by the image processing apparatus 3 according to the first embodiment will be described in detail with reference to the drawings.

FIG. 6 is a flow chart showing the operation of the image processing method according to the first embodiment.

[Step S1]
A captured image (original image) captured by the camera 2 is input to the image input unit 31 of the image processing device 3 . Here, it is assumed that the analog meter 1 is displayed in the captured image from the camera 2 .

[Step S2]
The orientation information detection unit 32 detects the image area of the analog meter 1 in order to acquire the orientation information of the analog meter 1 displayed in the input captured image (original image), and detects the outer shape of the analog meter 1 4 . Find the coordinate values of the vertices.

Here, the XY coordinate values of the four vertices (point Am, point Bm, point Cm, and point Dm) detected by the posture information detection unit 32 are called posture information.

An example of a method for detecting the coordinate values of the four vertices representing the outer shape of the analog meter 1 from the captured image by the posture information detection unit 32 will be described below.

FIG. 7 is a flowchart showing a method of detecting four vertex coordinates by the posture information detection unit 32 according to the first embodiment, and FIG. FIG. 10 is an explanatory diagram for explaining a method of detecting vertex coordinates of .

Here, the camera 2 photographs the analog meter 1 from an oblique direction with respect to the analog meter 1, and the photographed image is displayed in a state where the posture of the analog meter 1 is deformed as shown in FIG. 8(A). Illustrate the case.

The posture information detection unit 32 detects four straight lines indicating sides of the outer frame of the analog meter 1 reflected in the captured image P (step S11).

Here, for example, the posture information detection unit 32 performs binarization processing on the original image in order to extract an area including the outer frame of the analog meter 1 from the original image from the camera 2, and converts the white dial into a white dial. The area may be distinguished from the area of the outer frame around the dial, or the area of the outer frame of the analog meter 1 and the area of the background image may be distinguished. Furthermore, the posture information detection unit 32 may perform Hough transform on the image to extract straight lines including the sides of the outer frame of the analog meter 1 .

Then, the posture information detection unit 32 combines two straight lines among the four straight lines detected in the captured image P, and detects the intersection of the two straight lines. Intersections are detected for each combination of different straight lines (step S12).

For example, assume that four straight lines VV', TT', WW', and UU' in FIG. 8A are detected. At this time, the intersection of the straight lines VV' and TT', the intersection of the straight lines TT' and WW', the intersection of the straight lines WW' and UU', and the intersection of the straight lines UU' and VV' are detected.

The posture information detection unit 32 treats the four intersections as four vertices, assigns indexes (labels) of the points Am, Bm, Cm, and Dm to each vertex (step S13), and coordinates the coordinates of each vertex and each label. are associated with each other, and stored in the storage unit 33 as orientation information in the original image (step S14).

For example, the point Am in FIG. 8A is labeled with an index "Am", and the coordinate values (Xam, Yam) of the index "Am" are associated with the label "Am". Similarly, for the other three vertices, the associated labels are associated with the coordinate values of the respective vertices. The posture information of FIG. 8B thus obtained is stored in the storage unit 33 .

[Step S3]
The projective transformation parameter calculation unit 34 sums the XY coordinate values of the four vertices detected by the posture information detection unit 32 and the XY coordinate values of the four reference points that constitute preset posture reference information. Projective transformation parameters are derived using the eight XY coordinate values.

Here, the projective transformation parameters can be uniquely derived from the coordinate values of the corresponding points when homogeneous coordinates are used and there are four sets of corresponding points between known transformation sources and transformation destinations.

For example, the projective transformation parameter calculation unit 34 uses the four XY coordinate values of points Ar, Br, Cr, and Dr forming the attitude reference information in FIG. Each of the four XY coordinate values of points Am, Bm, Cm, and Dm that constitute the posture information in the image is set as a conversion destination, and a projective transformation parameter that allows each point of the transformation source to be projectively transformed to each point of the transformation destination. Calculate

[Step S4]
The needle detection coordinate sequence projective transformation unit 35 uses the projective transformation parameters derived by the projective transformation parameter calculation unit 34 to projectively transform all the coordinate values of the pointer detection coordinate sequence 332 .

For example, the pointer detection coordinate string 332 illustrated in FIG. 5B is configured to have n coordinate values each indexed. converted.

[Step S5]
The pointer detection unit 36 acquires pixel values from the image (original image) captured by the camera 2 using the pointer detection coordinate sequence after projective transformation, and detects pointer detection pixels composed of the pixel values of each coordinate value. Create a value column 361 .

FIG. 9 is an explanatory diagram illustrating the relationship between the pointer detection pixel value string 361 and the pointer detection coordinate string 332 in the first embodiment.

The pointer detection pixel value string 361 has pixel values of each coordinate value in the original image for all indexes that constitute the pointer detection coordinate string 332 after projective transformation. The needle detection pixel value column 361 may also provide pointer angles associated with each index of the pointer detection coordinate column 332 .

Here, the above-mentioned "obtaining pixel values from the original image using the pointer detection coordinate string after projective transformation" means that the pointer detection unit 36 configures the pointer detection coordinate string after projective transformation. It means that the pixel value of each coordinate value is obtained from the original image.

In addition, since an index is assigned to each coordinate value forming the pointer detection coordinate sequence 332, the pointer detection unit 36 converts the pixel value of each coordinate value of the pointer detection coordinate sequence 332 after projective transformation into the original image. , and each pixel value is associated with the corresponding index to create pointer detection pixel value string 361 . In other words, the needle detection pixel value string 361 has the same number of indexes as the indexes forming the pointer detection coordinate string 332 , and each index of the pointer detection pixel value string 361 corresponds to the pointer detection coordinate string 332 . is in a corresponding relationship with each index of .

Therefore, the pointer detection pixel value string 361 associates the index of the pointer detection coordinate string 332, the pixel value of each coordinate value of the pointer detection coordinate string after projective transformation in the original image, and the pointer angle. information.

Note that the needle detection pixel value string 361 in FIG. 9 corresponds to the pixel value of each coordinate value after the projective transformation in the original image, but each index corresponds to each coordinate value after the projective transformation in the original image. Coordinate values may be associated with each other.

[Step S6]
The pointer detection unit 36 refers to all pixel values forming the pointer detection pixel value sequence 361 to calculate the angle of the pointer.

As a method for calculating the pointer angle, for example, the luminance value of each pixel is calculated for all pixel values constituting the pointer detection pixel value sequence 361, and the pixel having the minimum luminance value among all the luminance values is selected. It is also possible to obtain the index of . Thus, by referring to the pointer detection coordinate column 332 or the pointer detection pixel value column 361 in which the pointer angle is associated with each index, the pointer angle corresponding to the index can be derived.

Further, for example, the luminance value of each pixel is calculated for all the pixel values constituting the needle detection pixel value sequence 361, and each luminance value is binarized using a threshold value. Alternatively, the index of the central pixel of the large interval may be obtained. Thus, by referring to the pointer detection coordinate column 332 or the pointer detection pixel value column 361 in which the pointer angle is associated with each index, the pointer angle corresponding to the index can be derived.

Here, the reason for deriving the index with the minimum luminance value will be explained. For example, pixels in the original image where the pointer is located have pixel values (for example, brightness values) that are smaller than pixels where the pointer is not located. Therefore, the pointer detection unit 36 can specify the position of the pointer by selecting the pixel value sequence having the minimum pixel value from among the pixel value sequences forming the pointer detection pixel value sequence.

Furthermore, as described above, the index of the pointer detection pixel value string 361 and the index of the pointer detection coordinate string 332 are in a corresponding relationship, so that the position of the pointer can be determined without requiring complicated coordinate conversion processing. If the pixel value of the indicated pixel can be specified and the index corresponding to that pixel value can be specified, the pointer angle corresponding to that index can be derived. In other words, the needle detection pixel value sequence 361 is composed of at least data paired with an index and a pixel value. Once identified, the pointer angle corresponding to the index can be derived.

[Steps S7 and S8]
The indicated value calculator 37 refers to the pointer detected coordinate column 332 and calculates the indicated value of the pointer based on the pointer angle detected by the pointer detector 36 . Then, the output unit 38 outputs information including the indicated value to the subsequent stage.

For example, the indicated value calculation unit 37 has a table that associates pointer angles and indicated values as illustrated in FIG. 23, and derives indicated values corresponding to the detected pointer angles. can be More specifically, for example, when the analog meter 1 is a voltmeter or the like, it has a table that associates pointer angles and voltage values, and voltage values (measured values) corresponding to pointer angles are stored. It may be derived as an indication value.

The method of deriving the indicated value of the pointer based on the pointer angle detected by the pointer detection unit 36 is not limited to the method described above.

(A-3) Effect of the First Embodiment As described above, the image processing apparatus of the first embodiment provides the coordinate values of four points based on the shape of the analog meter reflected in the input captured image , a projective transformation parameter is derived based on the coordinate values of the four reference points constituting the posture reference information, and projective transformation is performed on all the coordinate values (coordinate information) of the pointer detection coordinate string using the projective transformation parameter. do. Then, using all the coordinate values of the pointer detection coordinate string after projective transformation, the pixel values can be obtained from the original image, and the indicated value of the pointer can be read.

Further, the image processing apparatus according to the first embodiment uses the projective transformation parameter to projectively transform only the pointer detection coordinate string, so that a captured image containing a large amount of information such as brightness information and color information can be processed. Compared to projective transformation, the amount of information to be processed is small. Therefore, compared to the process of correcting the captured image such that the shape of the analog meter reflected in the input captured image approaches the ideal shape, the coordinate conversion process performed by the image processing apparatus of the first embodiment is performed by the process. requires less computation and memory usage.

Furthermore, in the image processing apparatus of the first embodiment, since the index of the pointer detection pixel value string and the index of the pointer detection coordinate string are in a corresponding relationship, the index of the pointer detection pixel value string and the index of the pointer detection coordinate string do not require complicated coordinate conversion processing. If the pixel value of the pixel indicating the position of the needle can be specified and the index corresponding to that pixel value can be specified, the pointer angle corresponding to that index can be derived. As a result, the amount of calculation and the amount of memory used for the analysis are small compared to the method of analyzing the entire captured image.

In addition, since the image processing apparatus of the first embodiment specifies the position of the pointer based on the pixel value of the coordinate value corresponding to the index of the pointer detection pixel value sequence, the shape of the pointer is limited by the conventional technology. Therefore, there are many types of analog meters that can be applied.

Further, the image processing apparatus according to the first embodiment requires a small amount of calculation for coordinate conversion processing, and thus consumes little power for the processing. The image processing device uses a general-purpose arithmetic processing device such as a microcomputer, and is assumed to receive power from an internal power supply (not shown) such as a battery, but the image processing device of the first embodiment is applied. This will extend the life of the internal power supply.

(B) Second Embodiment Next, a second embodiment of the image processing apparatus, image processing program, image processing method, and information collection system according to the present invention will be described in detail with reference to the drawings.

(B-1) Basic concept of the second embodiment When a camera takes an image of an analog meter, depending on the operating environment, the analog meter or camera may be subject to vibrations from facility equipment, for example. be. In such a case, the positions of the analog meter and the camera may move slightly, making it impossible to take an image of the analog meter in an upright posture.

In addition, depending on the installation environment of the camera for photographing the analog meter, the viewpoint of the camera may be rotated from the beginning, and the camera may not be able to photograph the analog meter in an upright position. For example, as shown in FIG. 10(A), it is assumed that the camera is tilted with respect to the analog meter (in FIG. 10(A), the camera is tilted 90 degrees to the right). As shown in FIG. 10B, the analog meter may be reflected in a rotated state in the captured image.

In order to solve the above problems, there are methods such as using a camera that has a function to correct the orientation of the captured image according to the camera posture, or using a device to hold the camera posture. The introduction cost will increase.

Therefore, it is required to more efficiently detect the position and orientation of the analog meter reflected in the captured image, perform image processing on the captured image, and efficiently read the pointer indication value indicated by the pointer.

(B-2) Configuration of Second Embodiment FIG. 11 is an internal configuration diagram showing the internal configuration of the image processing apparatus 3 according to the second embodiment.

In FIG. 11, an image processing apparatus 3A according to the second embodiment includes an image input unit 31, a posture information detection unit 32A, a storage unit 33, a projective transformation parameter calculation unit 34, a needle detection coordinate sequence projective transformation unit 35, an instruction It has a needle detection section 36 , an indicated value calculation section 37 and an output section 38 .

The image processing apparatus 10A of the second embodiment differs from that of the first embodiment in the posture information detection method by the posture information detection unit 32A. Since the other components that are the same as or correspond to the components described in the first embodiment can be applied, the components different from the first embodiment will be mainly described below.

In the second embodiment, a jig (hereinafter also referred to as "posture information detection jig") is attached to the analog meter 1 in order to detect the posture information of the analog meter 1 in the captured image (original image) of the camera 2. 50 is used. Then, after attaching this posture information detection jig 50 to a position surrounding the dial of the analog meter 1, the camera 2 photographs the analog meter 1, and the posture information detection unit 32A detects the posture information shown in the captured image. Posture information is detected by specifying the position of the feature point of the detection jig 50 .

FIG. 12 is a configuration diagram showing the configuration of the posture information detection jig 50 according to the second embodiment.

The posture information detection jig 50 has a substantially square shape as a whole. When the posture information detection jig 50 is viewed as a substantially square, the vertex neighborhood regions of the two sides forming one of the four vertices and the vertex neighborhood regions of the other three vertices are different from each other. configured to have color.

For example, an area near the vertex of the point A of the posture information detection jig 50 will be described. Region R11 and region R12 are formed in the same color. The region R11 has a curved upper left end 501 and a lower left end 502 that is linear. The region R12 has a curved upper left portion 503 and a straight upper right portion 504 . Thus, the point A is located at the boundary point between the regions R11 and R12.

Next, the region near the vertex of point B will be described as a representative of other vertices, i.e., points B, C, and D. FIG. The region R21 and the region R22 are formed with the same color. Regions R21 and R22 are regions of colors separated from the regions R11 and R12 on the color wheel when colors are expressed in the HSI color system. Region R21 has a curved upper right end portion 505 and a straight lower right end portion 506. Region R22 has a curved upper right portion 507 and a linear upper left portion 508. there is Point B is located at a boundary point between regions R21 and R22.

Further, for example, on the upper side of the substantially square posture information detection jig 50, there are regions R11 and R21 with different colors. This is because the regions near the vertex of the point B are configured to have different colors. When projective transformation is performed, the correspondence relationship between the transformation source points and the transformation destination points is obtained. Therefore, in this embodiment, of the four vertices of the posture information detection jig 50, in order to distinguish between the point A and the other points (point B, point C, and point D), the vertex of the point A is The colors of the neighborhood area and the vertex neighborhood area of other points are made different. The same applies to the left side of the posture information detection jig 50 .

FIG. 13 is an explanatory diagram illustrating a state in which the posture information detection jig 50 according to the second embodiment is attached to the analog meter 1. FIG.

The attitude information detection jig 50 is attached to the outer frame 111 of the analog meter 1 so as to surround the dial on the display surface of the dial of the analog meter 1 . At this time, the orientation information detection jig 50 is arranged so that the center of the orientation information detection jig 50 and the rotation center of the pointing needle are on the same axis, and the center of the orientation information detection jig 50 is aligned with the orientation. The posture information detection jig 50 is arranged so that the point A of the information detection jig 50 and the position of the scale "0" of the analog meter 1 are aligned on the straight line L1.

(B-3) Operation of Second Embodiment (B-3-1) Overall Operation Next, the operation of image processing by the image processing apparatus 3A according to the second embodiment will be described in detail with reference to the drawings. do.

FIG. 14 is a flow chart showing the operation of image processing by the image processing apparatus 3A according to the second embodiment.

[Step S1]
A captured image (original image) captured by the camera 2 is input to the image input unit 31 of the image processing device 3 .

Here, the attitude information detection jig 50 of FIG. 12 is attached to the analog meter 1, and the analog meter 1 to which the attitude information detection jig 50 is attached is photographed in the captured image to be input. It is assumed that there is It is also assumed that the format of the image captured by the camera 2 has color information, such as the RGB format.

[Step S20]
The posture information detection unit 32A detects the posture information detection jig 50 from the input captured image (original image), and detects four vertices (point Am, point Bm, point Cm, point Dm) to obtain the XY coordinate values.

Here, the XY coordinate values of the four vertices (point Am, point Bm, point Cm, and point Dm) detected by the posture information detection unit 32A are called posture information. A detailed description of the method for detecting the orientation information detection jig 50 from the original image will be given later.

[Step S3]
The projective transformation parameter calculator 34 configures the XY coordinate values of the four vertices (point Am, point Bm, point Cm, and point Dm) detected by the posture information detection unit 32A and preset posture reference information. Projective transformation parameters are derived using a total of eight XY coordinate values including the XY coordinate values of four reference points (point Ar, point Br, point Cr, and point Dr).

FIG. 15 is an explanatory diagram for explaining projective transformation with the reference points Ar, Br, Cr, and Dr of the orientation reference information as the conversion sources and the points Am, Bm, Cm, and Dm as the conversion destinations.

As shown in FIG. 15, in the second embodiment, similarly to the first embodiment, for example, the four XY coordinates of points Ar, Br, Cr, and Dr forming the attitude reference information in FIG. Each of the four XY coordinate values of points Am, Bm, Cm, and Dm constituting the orientation information detected from the original image by the orientation information detection unit 32A is assumed to be the conversion destination. to each point of the transformation destination.

[Step S4]
The pointer detection coordinate string projective transformation unit 35 uses the projective transformation parameters derived by the projective transformation parameter calculation unit 34 to projectively transform all the coordinate values of the pointer detection coordinate sequence before projective transformation.

In the second embodiment as well, projective transformation is performed on all the coordinate values that make up the needle detection coordinate sequence in FIG. 5B of the first embodiment.

[Step S5]
As in the first embodiment, the pointer detection unit 36 acquires pixel values from the captured image (original image) using all the coordinate values forming the pointer detection coordinate string after projective transformation, A needle detection pixel value sequence 361 is created.

As illustrated in FIG. 9, the needle detection pixel value string 361 has a structure in which a plurality of pieces of the data are linked, with each unit of data being a pair of an index and a pixel value. The number of data in the pointer detection pixel value column 361 is the same as the number of data in the pointer detection coordinate column 332 . The indices of the needle detection pixel value column 361 and the indices of the needle detection coordinate column 332 are in correspondence with each other.

[Step S6]
The pointer detection unit 36 refers to all pixel values forming the pointer detection pixel value sequence 361 to calculate the angle of the pointer.

In the second embodiment, similarly to the first embodiment, for example, the brightness value of each pixel is calculated for all pixel values that constitute the needle detection pixel value sequence 361, and each brightness value is calculated using a threshold value. may be binarized, and the index of the pixel having the minimum value among all luminance values may be obtained. As a result, the pointer angle corresponding to the index can be derived by referring to the pointer detection coordinate column 332 or the pointer detection pixel value column 361 in which the pointer angle is associated with each index. Further, similarly to the first embodiment, for example, the luminance value of each pixel is calculated for all the pixel values constituting the needle detection pixel value sequence 361, and each luminance value is binarized using a threshold to obtain a black color. Alternatively, the index of the pixel at the center of the section in which the number of consecutive pixels that are such that is the largest may be obtained.

Here, the reason for deriving the index with the minimum luminance value will be explained. For example, pixels in the original image where the pointer is located have pixel values (for example, brightness values) that are smaller than pixels where the pointer is not located. Therefore, the pointer detection unit 36 can specify the position of the pointer by selecting the pixel value sequence having the minimum pixel value from among the pixel value sequences forming the pointer detection pixel value sequence.

Furthermore, as described above, the index of the pointer detection pixel value string 361 and the index of the pointer detection coordinate string 332 are in a corresponding relationship, so that the position of the pointer can be determined without requiring complicated coordinate conversion processing. If the pixel value of the indicated pixel can be specified and the index corresponding to the coordinate value of that pixel can be specified, the pointer angle corresponding to that index can be derived.

[Steps S7 and S8]
The indicated value calculator 37 refers to the pointer detected coordinate column 332 and calculates the indicated value of the pointer based on the pointer angle detected by the pointer detector 36 . Then, the output unit 38 outputs information including the indicated value to the subsequent stage.

(B-3-2) Detection Method of Posture Information Detection Jig 50 FIG. 16 is a flowchart showing a detection method of the posture information detection jig 50 of the posture information detection unit 32A according to the second embodiment.

Here, the posture information detection jig 50 has a substantially square shape, and of the four vertices, two sides forming a point A, a region R11 and a region R12, are colored with “color A”, and the other three corners are colored. Region R21, region R22, region R23, and region R24 of each side forming vertices (point B, point C, and point D) are given "color B".

[Step S21]
The posture information detection unit 32A creates an image in which the color A applied to the regions R11 and R12 is emphasized (this is also called a "color A emphasized image") from the original image.

[Step S22]
The posture information detection unit 32A performs corner detection on the created color A-enhanced image. Here, for example, Harris corner detection can be used as the corner detection method.

[Step S23]
The posture information detection unit 32A selects one corner that is considered to be the most appropriate from the corners detected in step S22. The corner selection method will be described later.

[Step S24]
Subsequently, the posture information detection unit 32A creates an image in which the color B that forms the three vertices other than the point A among the four vertices of the posture information detection jig 50 is emphasized (hereinafter also referred to as a “color B emphasized image”). ) is created from the original image.

[Step S25]
The orientation information detection unit 32A performs corner detection on the created color B-enhanced image. For this corner detection method, for example, Harris' corner detection is used as in step S22.

[Step S26]
The posture information detection unit 32A selects three corners considered to be the most appropriate from the corners detected in step S25. The corner selection method will be described later.

[Step S27]
Subsequently, the posture information detection unit 32A labels the four points detected in steps S23 and S26.

Here, a label At is assigned to an arbitrary point among the four points, and labels are assigned in order of point Bt, point Ct, and point Dt in the clockwise direction starting from point At in the original image. A detailed description of labeling the four points will be given later.

[Step S28]
The orientation information detection unit 32A relabels the point formed by the area of the color A of the orientation information detection jig 50 so that the label is At.

At this time, each label is replaced so as to maintain the clockwise arrangement of the labels. The details of the method for determining whether each point is formed by color A or color B of the orientation information detection jig 50 will be described later. After that, At, Bt, Ct, and Dt are replaced with Am, Bm, Cm, and Dm, respectively, and the output is finished.

[Description of corner selection method]
Next, the corner selection method in steps S22 and S25 will be described. Here, points A, B, C, and D, which are four vertices of the posture information detection jig 50 appearing in the captured image (original image), are detected.

Point A is a boundary point between two regions with color A applied, and is located at a corner that is the intersection of two distinct edges. A can be detected.

Also, points B, C, and D are boundary points with two areas where color B is applied, and are positioned at corners that are intersections of two distinct edges, so that color B emphasis is performed. Point B, point C, and point D can be detected by selecting corners from the image.

First, the posture information detection unit 32A creates a binary image (corner binary image) in which pixels determined to be corners are white and pixels determined not to be corners are black.

An appropriate number of dilation operations are then performed on the corner binary image. The corner to be selected in this process is the vertex of the orientation information detection jig 50 . Here, each vertex of the orientation information detection jig 50 is distributed with sufficient distance in the image area. Therefore, corners that are very close to each other are considered to originate from one corner, so this dilation process is performed.

Subsequently, the orientation information detection unit 32A counts the number of white region clusters in the corner binary image subjected to the dilation processing. Subsequent processing changes depending on the number of lumps.

If the number of lumps is less than the target number, the threshold value for determining a corner in the corner detection process is loosened, and the corner detection process is retried, or the processing is restarted from re-capturing the original image.

Here, the target number means the number of corners to be detected. For example, in step S22, since one corner is detected from the color A-enhanced image, the target number is one in this case. Also, in step S25, since three corners are detected from the color B-enhanced image, the target number is three in this case.

On the other hand, when the number of chunks exceeds the target number, it is necessary to select the target number of chunks from the chunks. For example, the following chunk selection method can be used.

For example, consider a region with an appropriate radius centered on the barycentric coordinate value (hereinafter referred to as a lump region) in each lump region. The ratio of the color composition of the lump region in the original image is compared with the ratio of the color composition of the region having an appropriate radius around the vertex of the posture information detection jig 50 (hereinafter referred to as jig vertex region). Then, a higher required number is selected from those having similar ratios of color composition of the mass region and jig vertex regions.

In the ratio similarity determination, for example, the color ratio of each block region and jig vertex region is considered as a vector, and the cosine similarity is calculated with the vector of each block region and the vector of the jig vertex region, and the cosine similarity is A technique such as selecting a higher required number may be used.

Then, when the number of clusters matches the target number, one pixel, which is a corner, is selected from each cluster. For example, any method can be used as the method for selecting one pixel. For example, a method of selecting one pixel positioned at the center of gravity of the block region as a corner can be used.

[Detailed operation of step S27]
FIG. 17 is an explanatory diagram illustrating a method of labeling four vertices according to the second embodiment.

First, as illustrated in FIG. 17A, the posture information detection unit 32A assigns temporary labels (point At, point Bt, point Ct, and point Dt) to four points. Any method may be used to attach the label at this time.

Subsequently, the posture information detection unit 32A replaces the temporary labels given to the four points as follows.

First, it is determined whether or not the line segment At-Bt and the line segment Ct-Dt intersect. If they intersect, the labels of point At and point Dt are exchanged. For example, in FIG. 17B, the line segment At-Bt and the line segment Ct-Dt intersect, so the labels of the points At and Dt are interchanged.

Subsequently, it is determined whether or not the line segment Dt-At and the line segment Bt-Ct intersect. If they intersect, the labels of point At and point Bt are exchanged. For example, in FIG. 17C, since the line segment Dt-At and the line segment Bt-Ct do not intersect, the labels are not exchanged.

After that, it is determined whether or not the four points are arranged clockwise on the image. If they are arranged counterclockwise, the labels of points At and Ct are exchanged. For example, as shown in FIG. 17(C), when looking at the clockwise array with the point At as the starting point, the points At and Ct are not arranged in the order of point At→point Dt→point Ct→point Bt. replace the label of Then, as shown in FIG. 17D, the labels are arranged in order starting from the point At.

It should be noted that the specific means for judging the line segment intersection and the arrangement direction of the points described above is not limited to the method described above, and any method can be used as long as the labels can be arranged in order with the point At as the starting point. means may be used.

[Determination method in step S28]
Next, the operation of the determination method in step S28 will be described. This operation is similar to the blob selection method in the corner selection method described above.

First, consider the color ratio in the original image between an area with an appropriate radius centered on the coordinates of four vertices (hereafter referred to as vertex area) and an area formed by color A (hereafter referred to as color A area).

Considering the color ratio of each vertex region and the color ratio of the color A region as vectors, the cosine similarity is calculated between the vector of the vertex region and the vector of the color A region. Then, the point with the highest cosine similarity is selected as the point At.

(B-4) Effect of Second Embodiment As described above, according to the image processing apparatus of the second embodiment, the same effect as that of the first embodiment can be obtained.

Further, according to the image processing apparatus of the second embodiment, the orientation information detection jig is attached to the analog meter, the camera photographs the analog meter to which the orientation information detection jig is attached, and the image processing device detects the orientation information. Using the positions of the four vertices of the information detection jig, a projective transformation parameter indicating the difference between the ideal position and orientation of the analog meter reflected in the captured image is derived, and the pointer detection coordinates are calculated using the projective transformation parameter. Projectively transform all the coordinate values (coordinate information) of the column. Then, based on the pixel values of all the coordinate values of the pointer detection coordinate string after projective transformation, the pointer indication value can be read.

Since the second embodiment uses a substantially square orientation information detection jig made up of a combination of colors at distant positions on the color wheel, the hue difference between colors is sufficient. The positions of the four vertices can be easily derived. As a result, the projective transformation parameters can be easily derived based on the four vertices of the orientation information detection jig and the four reference points of the orientation information.

In addition, since the second embodiment employs a simple method of attaching the posture information detection jig to the analog meter, it is possible to reduce the introduction cost of the indicated value reading system. Furthermore, since the effects of the present invention can be fully exhibited if the surface of the dial of the analog meter and the orientation information detection jig are captured within the imaging range of the camera, the relative positions of the analog meter and the camera are Restrictions on relationships and camera orientation can be relaxed. Therefore, the degree of freedom in installing the camera with respect to the analog meter is increased.

(C) Third Embodiment Next, a third embodiment of the image processing apparatus, image processing program, image processing method, and information collection system according to the present invention will be described in detail with reference to the drawings.

(C-1) Configuration of the Third Embodiment In the third embodiment, as in the second embodiment, an attitude information detection jig is attached to an analog meter, and the attitude information detection unit detects the attitude from the captured image. The positions of the four vertices of the information detection jig are detected, and the projective transformation parameters are derived based on the coordinate values of the four vertices and the four reference points of the posture reference information.

However, in the third embodiment, the configuration of the orientation information detection jig attached to the analog meter and the processing of detecting the positions of the four vertices of the orientation information detection jig from the captured image by the orientation information detection unit are different from those of the second embodiment. Different from the embodiment.

Therefore, the image processing apparatus of the third embodiment will be described with a focus on the components different from those of the second embodiment, using FIG. 11 of the second embodiment.

FIG. 18 is a configuration diagram showing the configuration of an orientation information detection jig 50A according to the third embodiment.

The posture information detection jig 50A has a substantially square shape as a whole. Each side of the substantially square posture information detection jig 50A is composed of two colors. For the combination of colors on each side, which are composed of two colors, colors at distant positions on the color wheel are selected, for example, when colors are expressed in the HSI color system.

On each side composed of two colors, the boundary lines of the regions of each color are straight lines a, straight lines b, straight lines c, and straight lines d. The vertex formed by straight lines d and a is point A, the vertex formed by lines a and b is point B, the vertex formed by lines b and c is point C, and the vertex formed by lines c and d is point D. there is

Of the four sides, the two sides having the straight line b and the straight line c as boundary lines are combined with the same color. Therefore, each side of the orientation information detection jig 50A is formed by three sets of color combinations. Thus, it is desirable to use at least three color combinations.

FIG. 19 is an explanatory diagram illustrating a state in which the posture information detection jig 50A according to the third embodiment is attached to the analog meter 1. FIG.

The method of attaching the attitude information detection jig 50A to the analog meter 1 is basically the same as in the second embodiment. That is, the posture information detection jig 50A is provided on the outer frame 111 of the analog meter 1 so as to surround the dial of the analog meter 1 .

At this time, the orientation information detection jig 50A is arranged so that the center of the orientation information detection jig 50A and the rotation center of the pointing needle are on the same axis, and the center of the orientation information detection jig 50A is aligned with the orientation of the orientation information detection jig 50A. The posture information detection jig 50 is arranged so that the point A of the information detection jig 50A and the position of the scale "0" of the analog meter 1 are aligned on the straight line L1.

(C-2) Operation of the Third Embodiment The image processing method of the image processing apparatus 3A according to the third embodiment is basically described in the second embodiment using the flowchart of FIG. Same as method.

Therefore, in the third embodiment, processing for detecting the positions of the four vertices of the posture information detection jig 50A by the posture information detection unit 30A will be described in detail.

FIG. 20 is a flowchart showing a detection method of the posture information detection jig 50 of the posture information detection section 32A according to the third embodiment.

[Step S31]
The posture information detection unit 32A performs HSI conversion on the captured image (original image) input from the image input unit 11 . As a result, the captured image can be an image composed of the three components of hue, saturation, and luminance.

[Steps S32 to S37]
In steps S32 to S37, among the three components of hue, saturation, and luminance, attention is paid to the hue (H component) of each pixel, and an H component difference image is created. Here, among all the pixels of the original image, a certain pixel is set as a target pixel and the following processing is performed, and the processing of steps S32 to S37 is repeated for all pixels.

[Step S33]
The posture information detection unit 32A sets a certain pixel as a target pixel in the original image, and calculates the absolute value of the difference between the H component value of the target pixel and the H component values of the eight surrounding pixels of the target pixel. As a result, eight H-component absolute difference values are obtained for one pixel of interest.

[Step S34]
The posture information detection unit 32A determines the magnitude relationship of the eight H-component difference absolute values for the pixel of interest, and sets the maximum value as the H-component difference value of the pixel of interest.

[Step S35]
Next, the posture information detection unit 32A acquires the value of the saturation (S component) of the pixel of interest, and compares the value of the S component with a threshold. Then, if the value of the S component of the pixel of interest is less than the threshold, the process proceeds to step S36. If the value of the S component of the pixel of interest is not less than the threshold, the process proceeds to step S37.

[Step S36]
When the S component value of the target pixel is less than the threshold value, the H component difference value of the target pixel is set to zero.

[Step S37]
The posture information detection unit 32A performs the processing of steps S32 to S37 on all pixels of the captured image (original image) to create an H component difference image. Therefore, if the processing has not been completed for all pixels, the processing of steps S32 to S37 is performed for all pixels.

[Step S38]
The orientation information detection unit 32A applies a Hough transform to the H component difference image to detect four straight lines L11, L12, L13, and L14 forming the orientation information detection jig 50A. Each of these four straight lines corresponds to a boundary between two colors on each side of the orientation information detection jig 50A.

[Step S39]
The posture information detection unit 32A detects four intersections using the detected four straight lines.

Specifically, one of the four straight lines is used as a reference straight line, and the absolute value of the difference between the slope of the reference straight line and the slopes of the other three straight lines is calculated. Then, two straight lines having a large absolute value of the slope difference are selected. A point of intersection between the reference straight line and the two straight lines is calculated. Two intersection points are obtained as a result.

When the above-described processing is performed for each of the four straight lines and duplication of coordinate values obtained thereby is eliminated, four coordinate values are obtained.

[Step S40]
The posture information detection unit 32A labels the four straight lines detected in step S38 as straight lines a, straight lines b, straight lines c, and straight lines d. Two of the intersections detected in step S39 are present on a certain straight line among the four straight lines. From the value of the H component of the pixels in the vicinity of the line segment connecting the two intersections in the original image, which straight line of the posture information detection jig 50A is identified and labeled.

[Step S41]
The posture information detection unit 32A labels the four intersection points detected in step S39 as point A, point B, point C, and point D. FIG.

During labeling, the label of the straight line labeled in step S40 is used to determine which two straight lines intersect.

(C-3) Effects of Third Embodiment As described above, according to the image processing apparatus of the third embodiment, effects similar to those of the first and second embodiments can be obtained.

(D) Fourth Embodiment Next, a fourth embodiment of the image processing apparatus, image processing program, image processing method, and information collection system according to the present invention will be described in detail with reference to the drawings.

A case where the fourth embodiment is applied to the second or third embodiment will be exemplified below, but similar effects can be obtained when the fourth embodiment is applied to the first embodiment. .

FIG. 21 is an internal configuration diagram showing the internal configuration of an image processing apparatus according to the fourth embodiment.

21, an image processing apparatus 3B of the fourth embodiment includes an image input unit 31, a posture information detection unit 32A, a storage unit 33, a projective transformation parameter calculation unit 34B, an original image projective transformation unit 39, and a pointer detection unit 36B. , an indicated value calculator 37 and an output unit 38 .

In the fourth embodiment, each process of the projective transformation parameter calculation unit 34B and pointer detection unit 36B, and the pointer detection coordinate sequence projective transformation unit 35 of the first to third embodiments are replaced by the original image projective transformation unit 39. , which is different from the first to third embodiments.

The projective transformation parameter calculation unit 34B calculates the XY coordinate values of four vertices Am, Bm, Cm, and Dm detected from the captured image (original image), and the four reference points Ar, Br, Similar to the first to third embodiments, the projective transformation parameters are calculated based on the XY coordinate values of eight points of the XY coordinate values of Cr and Dr. It differs from the first to third embodiments.

That is, the projective transformation parameter calculation unit 34B uses the four vertices Am, Bm, Cm, and Dm extracted from the captured image (original image) as the transformation sources, and calculates the posture reference information 331. Projection transformation is performed using the four reference points Ar, Br, Cr, and Dr as transformation destinations. As described above, the coordinate values used as the conversion source and the conversion destination are different from those in the first to third embodiments.

The original image projective transformation section 39 projects the captured image (original image) using the projective transformation parameter calculated by the projective transformation parameter computing section 34B.

The pointer detection unit 36B refers to each coordinate value forming the pointer detection coordinate string 332, acquires pixel values from the original image that has been projectively transformed by the original image projective transformation unit 39, and converts the pointer detection pixel value string to is created.

That is, in the first to third embodiments, the projective transformation parameter is used to projectively transform each coordinate value constituting the pointer detection coordinate string, and the pixel value of the corresponding coordinate value in the original image is obtained. However, the pointer detection unit 36B of the fourth embodiment calculates the pixel values of each coordinate value forming the pointer detection coordinate string from the original image after the projective transformation. Acquire and create a needle detection pixel value string.

(D-2) Operation of Fourth Embodiment FIG. 22 is a flow chart showing image processing in the image processing apparatus 3B according to the fourth embodiment.

[Steps S1, S20]
A captured image (original image) captured by the camera 2 is input to the image input unit 31 of the image processing device 3 (step S1). The detection jig 50 is detected, and the XY coordinate values of the four vertices (point Am, point Bm, point Cm, and point Dm) of the orientation information detection jig 50 in the image are obtained (step S20).

[Step S50]
The projective transformation parameter calculation unit 34 uses the XY coordinate values of the four vertices (point Am, point Bm, point Cm, and point Dm) detected by the posture information detection unit 32 as the source of transformation, and converts the four vertices constituting the posture reference information. Using the XY coordinate values of the reference points (point Ar, point Br, point Cr, point Dr) as conversion destinations, projective transformation parameters are calculated so that the transformation source points can be projectively transformed to the transformation destination points.

[Step S51]
The original image projective transformation unit 39 projects the captured image (original image) using the projective transformation parameter calculated by the projective transformation parameter computing unit 34B.

[Step S52]
Then, the pointer detection unit 36B refers to each coordinate value forming the pointer detection coordinate string 332, acquires a pixel value from the original image that has been projectively transformed by the original image projective transformation unit 39, and obtains a pixel value from the pointer detection pixel. Create a value column.

[Steps S6 to S8]
Since the processing of steps S6 to S8 is the same as in the first to third embodiments, detailed description thereof will be omitted.

(D-3) Effects of the Fourth Embodiment As described above, the same effects as those of the first to third embodiments can be obtained with the image processing apparatus of the fourth embodiment.

(E) Other Embodiments (E-1) In the first to third embodiments described above, a plurality of pixel values extracted from a captured image are detected by referring to the projection-transformed pointer detection coordinate string. I exemplified that. Further, in the fourth embodiment, the case where the pointer position is detected from a plurality of pixel values extracted from the captured image that has undergone projective transformation using the pointer detection coordinate sequence has been exemplified.

At this time, when detecting the position of the pointer from the pixel values of each coordinate value of the pointer detection coordinate row, the pixel values of a certain area including each coordinate value of the pointer detection coordinate row are considered, and each coordinate value is included. The pointer position may be detected from pixel values in a certain area. This is because, for example, as shown in FIG. 24A, numbers such as "0" are assigned to the dial, and in that case, the pixel value of the coordinate value in the captured image becomes small. There is In such a case, it can be avoided by setting the position of the virtual circle VC2 for arranging each coordinate value of the needle detection coordinate string at a position that does not overlap the numbers on the dial. However, if the virtual circle VC2 were to overlap the numbers on the dial, as shown in FIG. Pixel values may also be considered.

(E-2) In the first to fourth embodiments described above, the calculation of the projective transformation parameters is expressed as if the projective transformation parameters are calculated each time an image captured by the camera 2 is input. The projective transformation parameter calculation process may be performed at the time of initial setting of the image processing system, or may be performed at predetermined time intervals.

That is, the projective transformation parameter is not limited to be calculated each time the image processing is performed, but the projective transformation parameter calculated once is stored, and using the projective transformation parameter, the pointer detection coordinate string or The original image may be subject to projective transformation. This makes it possible to reduce the processing load associated with the calculation of the projective transformation parameters. Of course, the projective transformation parameters may be calculated for each image processing.

(E-3) In the second to fourth embodiments described above, when the posture information detection jig is attached to the analog meter, the posture information detection jig is attached so as to surround the range in which the pointer of the analog meter moves. exemplified. However, the position where the orientation information detection jig is attached is not limited to this.

The orientation information detection jig uses the XY coordinate values of the four vertices of the orientation information detection jig shown in the captured image and the XY coordinate values of the four reference values of the orientation reference information to obtain the projection transformation parameters. contributes to the derivation of Therefore, the mounting position is not particularly limited as long as the posture information detection jig is shown together with the pointer of the analog meter in the captured image.

In this case, it should be noted that the XY coordinate values of the four vertices forming the posture information detection jig are determined in consideration of the diameter and position of the circle (virtual circle VC2) formed by the pointer detection coordinate string. Also, when photographing with a camera, it is necessary that the movable range of the posture information detection jig and the pointer of the analog meter are all within the photographing range, so this also requires attention.

(E-4) In the above-described second to fourth embodiments, the posture information detection jig has a substantially square shape, but the shape of the posture information detection jig may be an arbitrary square. In this case, it is necessary to determine the coordinates of the four vertices that form an arbitrary quadrangle by considering the diameter and position of the circle that the pointer detection coordinate row forms.

(E-5) In the above-described first to fourth embodiments, the case where the pointer is an analog meter that rotates is exemplified, and the coordinate group for detecting the position of the pointer (pointer detection coordinate row) is A case of existing on the virtual circle VC2 is illustrated. However, each coordinate value of the pointer detection coordinate row may be arranged on a straight line, or may have any other shape. In this case, it should be noted that the coordinates of the four vertices forming the orientation information detection jig are determined in consideration of the shape formed by the coordinate group.

For example, as shown in FIG. 25A, it may be a horizontal instrument in which the pointer moves in the horizontal direction. In this case, as shown in FIG. The values may be arranged on a straight line in the horizontal direction (X-axis direction) according to the movement of the pointer.

In addition, it may be a vertical instrument in which the pointer moves in the vertical direction. They may be arranged on a straight line.

(E-6) In the above-described second to fourth embodiments, when attaching the posture information detection jig to the analog meter, the center of the posture information detection jig and the rotation center of the pointer are aligned on the same axis. , An example of attaching the posture information detection jig to the analog meter so that the center of the posture information detection jig, the point A of the posture information detection jig, and the position of the scale "0" of the analog meter are aligned on a straight line. bottom. However, the arrangement is not limited to such an arrangement, and the orientation is such that the center of the orientation information detection jig, the point A of the orientation information detection jig, and the scale "0" of the analog meter are aligned on a straight line. The information detection jig may not be attached. In this case, it is necessary to provide the image processing apparatus with the angle formed by the point A of the orientation information detection jig, the center of the orientation information detection jig, and the scale "0" of the analog meter.

DESCRIPTION OF SYMBOLS 10... Information collection system 1, 1A... Analog meter, 2... Camera, 3, 3A, 3B... Image processing apparatus, 4... Communication apparatus, 5... Communication apparatus, 6... Management apparatus, 31... Image input part, 32, 32A Posture information detection unit 33 Storage unit 331 Posture reference information 332 Pointer detection coordinate sequence 34, 34B Projection transformation parameter calculation unit 35 Pointer detection coordinate sequence projective transformation unit 36, 36B Pointer detection unit 37 Pointed value derivation unit 38 Output unit 39 Original image projective transformation unit 361 Pointer detection pixel value string 50, 50A Posture information detection jig.

Claims (9)

An image of a display surface of a pointer instrument having a plurality of vertices and having, as a feature point , a vertex distinguishable from other vertices among the plurality of vertices, the display surface of the pointer instrument to which a posture information detection jig is attached is obtained from a photographed image of the display surface of the pointer instrument. In the image processing device that derives the indicated value of
Using a projective transformation parameter between a virtual reference image of the pointer gauge and an image of the pointer gauge in the input captured image, the position of the pointer of the pointer gauge is indicated in the reference image. a pointer detection coordinate sequence having a plurality of coordinate values, or a projective transformation unit that projectively transforms the captured image;
From a plurality of pixel values extracted from the captured image using the pointer detection coordinate sequence that has undergone projective transformation, or from a plurality of pixel values extracted from the captured image that has undergone a projective transformation using the pointer detection coordinate sequence , a pointer detection unit for detecting the position of the pointer;
an indicated value derivation unit for deriving the indicated value corresponding to the position of the pointer detected by the pointer detection unit;
Coordinate values of the plurality of vertices of the posture information detection jig appearing in the captured image are detected from the input captured image, and the posture of the pointer instrument is specified based on the positions of the feature points. a posture information detection unit that
A projective transformation parameter derivation unit for deriving the projective transformation parameter based on each coordinate value of the plurality of reference points forming the reference image and each coordinate value of the plurality of vertices detected by the posture information detection unit. An image processing device comprising: The pointer detection coordinate sequence is
having the plurality of coordinate values provided corresponding to the dial of the indicator needle within the area in which the pointer exists in the reference image;
2. The image processing apparatus according to claim 1, wherein at least the plurality of coordinate values are associated with the position of the pointer in the pointer instrument. When the projective transformation unit projectively transforms the pointer detection coordinate string using the projective transformation parameter,
The pointer detection unit refers to each of the coordinate values after projective transformation of the pointer detection coordinate string, and selects the minimum pixel value from among the plurality of pixel values extracted from the captured image. 3. The image processing device according to claim 1, wherein the position of the pointer is detected. wherein the pointer detection coordinate string is further provided with identification information for identifying each of the coordinate values for each of the plurality of coordinate values,
The pointer detection unit associates a plurality of pixel values extracted from the captured image with reference to the coordinate values after projective transformation, the identification information, and the position of the pointer on the pointer instrument. 3. The image processing according to claim 1, wherein the position of the pointer corresponding to the minimum pixel value is detected from among the plurality of pixel values by referring to the pointer detection pixel value sequence. Device. When the projective transformation unit projectively transforms the captured image using the projective transformation parameter,
The pointer detecting unit refers to each of the coordinate values of the pointer detection coordinate string, and out of a plurality of pixel values extracted from the captured image after projective transformation, the pixel value corresponding to the minimum pixel value. 3. The image processing device according to claim 1, wherein the position of the pointer is detected. wherein the pointer detection coordinate string is further provided with identification information for identifying each of the coordinate values for each of the plurality of coordinate values,
A pointer detection pixel value sequence in which a plurality of pixel values extracted from the captured image after projective transformation by the pointer detection unit, the identification information, and the position of the pointer in the pointer instrument are associated with each other is referred to. 3. The image processing apparatus according to claim 1, wherein the pointer position corresponding to the minimum pixel value is detected from among a plurality of pixel values. An image of a display surface of a pointer instrument having a plurality of vertices and having, as a feature point, a vertex distinguishable from other vertices among the plurality of vertices, the display surface of the pointer instrument to which a posture information detection jig is attached is obtained from a photographed image of the display surface of the pointer instrument. In the image processing program that derives the indicated value of
the computer,
Using a projective transformation parameter between a virtual reference image of the pointer gauge and an image of the pointer gauge in the input captured image, the position of the pointer of the pointer gauge is indicated in the reference image. a pointer detection coordinate sequence having a plurality of coordinate values, or a projective transformation unit that projectively transforms the captured image;
From a plurality of pixel values extracted from the captured image using the pointer detection coordinate sequence that has undergone projective transformation, or from a plurality of pixel values extracted from the captured image that has undergone a projective transformation using the pointer detection coordinate sequence , a pointer detection unit for detecting the position of the pointer;
an indicated value derivation unit for deriving the indicated value corresponding to the position of the pointer detected by the pointer detection unit;
Coordinate values of the plurality of vertices of the posture information detection jig appearing in the captured image are detected from the input captured image, and the posture of the pointer instrument is specified based on the positions of the feature points. a posture information detection unit that
A projective transformation parameter derivation unit for deriving the projective transformation parameter based on each coordinate value of the plurality of reference points forming the reference image and each coordinate value of the plurality of vertices detected by the posture information detection unit. An image processing program characterized by functioning as An image of a display surface of a pointer instrument having a plurality of vertices and having, as a feature point, a vertex distinguishable from other vertices among the plurality of vertices, the display surface of the pointer instrument to which a posture information detection jig is attached is obtained from a photographed image of the display surface of the pointer instrument. In the image processing method for deriving the indicated value of
A projective transformation unit uses projective transformation parameters between a virtual reference image of the pointer gauge and an image of the pointer gauge in the input captured image to indicate the pointer gauge with the reference image. Projectively transforming the pointer detection coordinate string having a plurality of coordinate values indicating the position of the needle or the captured image,
A pointer detection unit extracts a plurality of pixel values extracted from the captured image using the pointer detection coordinate string that has undergone projective transformation, or extracts from the captured image that has undergone projective transformation using the pointer detection coordinate sequence. detecting the position of the pointer from the plurality of pixel values obtained,
an indicated value derivation unit deriving the indicated value corresponding to the position of the pointer detected by the pointer detection unit;
A posture information detection unit detects each coordinate value of the plurality of vertexes of the posture information detection jig appearing in the captured image from the input captured image, and performs the instruction based on the positions of the feature points. Identify the posture of the needle instrument ,
A projective transformation parameter derivation unit calculates the projective transformation parameter based on each coordinate value of a plurality of reference points forming the reference image and each coordinate value of each of the plurality of vertices detected by the posture information detection unit. An image processing method characterized by deriving. An image of a display surface of a pointer instrument having a plurality of vertices and having, as a feature point, a vertex distinguishable from other vertices among the plurality of vertices, the display surface of the pointer instrument to which a posture information detection jig is attached is obtained from a photographed image of the display surface of the pointer instrument. an image processing device for deriving an indication value of
a first communication device that transmits a signal including the indication value of the pointer gauge derived by the image processing device;
a second communication device that receives the signal transmitted from the first communication device;
a management device that stores the indicated value included in the signal received by the second communication device;
An information collection system, wherein the image processing device is the image processing device according to any one of claims 1 to 6.

Images