JP2010267113A - Component management method, device, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for managing the information of components specified by using an image photographed without receiving any restriction from photographic conditions. <P>SOLUTION: Replacement component management method is characterized to consist of: a component photographing step that portable terminal equipment photographs components with markers arranged at a prescribed position relation for components by an electronic camera, and transmits photographic image data to a component management device; a component CG image generating step that the component management device detects a marker image from the photographic image data, and calculates a coordinate conversion matrix by using the shape information of a standard marker, and generates a component CG image by applying the coordinate conversion matrix to three-dimensional CAD data; and a component information picture creation returning step that the component management device calculates similarity degree by comparing a component CG image with the component image of the photographic image data, and creates component information picture data by using the component data of the component CG image whose similarity is high, and returns the component information picture data to portable terminal equipment. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

The present invention relates to a method, an apparatus, a program, and a recording medium for managing component information using a component captured image.

The present invention specifies a replacement part by extracting a part having a similar shape as a replacement part candidate from many types of parts using an image obtained by taking a photograph of the part, particularly at a construction site or a maintenance shop. This is useful for improving work efficiency.

In order to maintain and manage facilities such as buildings and machinery products such as automobiles, it is indispensable to repair damaged parts and maintain them by regular inspections.
In order to facilitate smooth repairs and maintenance at the construction site of a facility or a machine product maintenance factory, it is important to quickly identify parts that have failed or have been regularly inspected and arrange replacement parts.

Therefore, a method for specifying a replacement part will be described.
<< A. Part identification by part number >>
As a method for specifying a replacement part, a method using the part number of the replacement part is generally used.
The part number is checked by reading what is written on the repair part or referring to the part catalog.
For example, Patent Document 1 discloses a technique for sending a replacement part by displaying an image of a part specified by a part number on a terminal when the part number is input to a terminal connected to the Internet. Yes.

<< B. Part identification by image search >>
Alternatively, as a method for specifying a replacement part, there is a method for searching and specifying a replacement part by searching a part database using an image of the replacement part as a search key.
For example, Patent Document 2 discloses a technique for determining (= identifying) a product by inputting a handwritten image of the product and using it as a search key and comparing it with a product image in a product database.
Alternatively, in Non-Patent Document 1, a part is photographed with a mobile phone, feature data of the part photographed image is obtained, and this is transmitted to a corresponding website on the Internet, and a response corresponding to the feature data is received. A technique for displaying this on a mobile phone is disclosed.

JP 2001-256382 A (paragraph number “0023” to paragraph number “0032”, FIG. 3) JP 2006-209260 A (paragraph number “0071” to paragraph number “0079”, FIG. 9)

Olympus Corporation New Business Planning Department, “Sync ★ R”, Service Overview-Features (Outline of Sync ★ R), Olympus Corporation, [online], [Search March 2, 2009], Internet <Http://gwmj.jp/>

However, << A. In the method of “part identification by part number”, there is a problem that it takes a lot of time to search for a desired part by searching for a desired part from a large amount of part information in the part catalog. In addition, when the part number written on a failure or damaged equipment / part is unclear, it may not be read. Or, it may not be listed in the parts catalog because the equipment / parts damaged or damaged are old.
In addition, << B. In the method of specifying a part by image search, when an image search is performed using a handwritten image, if the handwritten image does not capture the characteristics of the part, the desired product cannot be searched and the part cannot be specified. is there. Alternatively, when searching for an image using a photographed image, correct feature data cannot be generated unless the part is photographed under a photographing condition such as a predetermined photographing angle. There arises a disadvantage that information (for example, information for specifying a part) cannot be obtained.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and an object of the present invention is to manage a component information specified by using an image captured without being restricted by an imaging condition. , Providing an apparatus, a program, and a recording medium.

This invention solves the said subject by the means as described in each following aspect.
That is, the first invention of the present invention is:
A replacement part management method using a portable terminal apparatus with an electronic camera and a part management apparatus having an image recognition program,
In a replacement part management method using a part database for managing standard marker information, three-dimensional CAD data and part data associated with a part number, provided in the part management apparatus,
A component photographing step of photographing a component and generating photographed image data together with a marker arranged in a predetermined positional relationship with respect to the component by an electronic camera of the mobile terminal device;
A captured image transmission step of transmitting the captured image data to the parts management device;
A marker detection step in which the component management device detects a marker image from the received captured image data;
An imaging condition calculation step in which the component management device calculates a coordinate transformation matrix using the detected marker image and the shape information of the standard marker,
The component management apparatus applies the calculated coordinate transformation matrix to the three-dimensional CAD data read from the component database, performs rendering processing, generates a component CG image, and associates the component number with the component CG image generation step. When,
The component management device compares the component CG image data generated using the image recognition program with the component image of the photographed image data, calculates the similarity, and determines the component number associated with the component CG image with a high similarity. A component identification step to be extracted;
A parts information screen creation reply step in which the parts management device creates part information screen data using the part data of the extracted part number and sends it back to the mobile terminal device;
This is a replacement part management method characterized by being performed in a procedure including

In this way, by arranging the markers in a predetermined positional relationship with respect to the component, it is possible to calculate the imaging conditions for the component.
In addition, it is possible to easily and easily extract part data of parts similar to the shape of the photographed part by recognizing the part photographed image using the part CG image generated using the part photographing condition. It is.
Here, the marker is a square figure having a pattern that is not point-symmetric. Since the dimensions and pattern of the marker are known, the shooting distance and shooting angle of the camera can be calculated from the size of the photographed marker shape and the degree of deformation. Here, since the marker uses a symbol that is not point-symmetric, even if the marker rotates 180 degrees, the same symbol is not obtained.

The second invention of the present invention is a standard marker storage area for storing standard marker information;
Parts information screen storage area for storing parts information screen data,
A component storage area for storing a three-dimensional CAD data associated with a component number, a component database having an image recognition program for managing the component data associated with the component number;
A captured image storage area for storing captured image data including a component image and a marker image;
Storage means having
A photographed image receiving means for receiving photographed image data obtained by photographing a component together with a marker arranged in a predetermined positional relationship with respect to the component;
Marker detection means for detecting a marker image from received photographed image data;
An imaging condition calculation means for calculating a coordinate transformation matrix using the detected marker image and dimension information of the standard marker,
A component CG image generation unit that generates a component CG image by applying the coordinate transformation matrix calculated by the imaging condition calculation unit to the three-dimensional CAD data read from the component database, generates a component CG image, and associates it with the component number When,
A component that specifies a component number associated with a high-similarity component CG image by calculating the similarity between the generated component CG image data and the component image of the captured image data with respect to a component database having an image recognition program Specific means,
A part data extraction means for searching by using the specified part number as a search key and receiving the extracted part data;
Part information screen creation means for creating part information screen data using the extracted part data;
A component information screen return means for returning the created component information screen data to the mobile terminal device;
A component management device comprising:

A third invention of the present invention is a computer program that causes a computer to operate as the component management device according to the second invention by being incorporated in the computer.

A fourth invention of the present invention is a computer-readable recording medium on which the computer program according to the third invention is recorded.

According to the present invention,
(1) By using a marker arranged in a predetermined positional relationship with respect to a part, it is possible to photograph the part without being restricted by photographing conditions.
(2) By recognizing the captured image of the component using the component CG image generated by rendering the component CG image under the same shooting conditions (= the same shooting angle, the same shooting distance, etc.) as the captured image of the component. It is possible to extract replacement part candidates similar to the shape of the photographed part.
Therefore, by using an image obtained by photographing a component together with a marker arranged at a predetermined position, there is an effect that it is possible to improve the efficiency of work for specifying a replacement component.

FIG. 1 shows the overall configuration of the component management system of the present invention. (Example 1) FIG. 2 is a diagram for explaining a rough work procedure of the component management system. FIG. 3 is an example of a display screen of the mobile terminal device. FIG. 4 is an example of a marker. FIG. 5 is a diagram illustrating the relationship between the coordinate systems. FIG. 6 is a diagram for explaining the component database. FIG. 7 is a diagram illustrating a detailed processing procedure of (4) << Calculation condition calculation >>. FIG. 8 is a diagram for explaining the detailed processing procedure of (5) << part data extraction >>. FIG. 9 is a diagram illustrating a detailed configuration diagram of the server apparatus. FIG. 10 is a diagram illustrating a detailed configuration diagram of the information processing apparatus. FIG. 11 is an example of a detailed processing procedure of (5) << part data extraction >>.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings.

FIG. 1 is a diagram illustrating an outline of a component management system 1 according to an embodiment of the present invention.
The component management system 1 includes a component information server device 100 (= component management device) and a mobile terminal device 300.

The mobile terminal device 300 is wirelessly connected to the base station 600 and is connected to the component information server device 100 on the network 900 via the gateway 800.
The mobile terminal device 300 is an existing mobile phone or smartphone, and includes an electronic camera and is loaded with a dedicated program described later.
The electronic camera of the mobile terminal device 300 generates two-dimensional digital image data having a gray scale or RGB value.

The component information server apparatus 100 is a server computer equipped with an existing database system, in which an image recognition program by a well-known technique and a dedicated program described later are installed.

FIG. 2 explains a rough work procedure of the parts management system 1.
(1) << Part photography >> The contractor photographs parts and markers with the electronic camera of the mobile terminal device 300.
(2) << Transmission of Captured Image >> The mobile terminal device 300 transmits captured image data (= two-dimensional digital image data) to, for example, an e-mail attached to the component management server device.
(3) << Received Captured Image >> The component information server apparatus 100 receives captured image data.
(4) << Calculation of shooting conditions >> The component information server device 100 calculates shooting conditions from the shape of the marker image of the received shot image.
(5) << Extraction of Part Data >> The part information server apparatus 100 generates part CG image data (= two-dimensional digital image data) generated by applying this photographing condition to the three-dimensional CAD data in the part database. Similarity is calculated in comparison with a component captured image cut out from the captured image data, and component data of a component CG image having a similarity equal to or higher than a predetermined threshold is extracted from the component database.
(6) << Reply of part information screen >> The part information server apparatus 100 creates part information screen data using the extracted part data and sends it back to the mobile terminal apparatus 500.
(7) << Display of Component Information Screen >> The band terminal device 300 receives and displays the component information screen data.
(8) << Replacement Parts Order >> When the construction contractor views the parts information screen displayed on the mobile terminal device 300 and inputs information for ordering a replacement product, the mobile terminal device 300 accepts this input and places an order. Information is transmitted to the component information server apparatus 100.

FIG. 4 is an example of the marker 16.
The marker 16 is a pattern having a figure in a square black thick frame.
For this figure, a two-dimensional code pattern, a pattern, a character, or the like that is not point-symmetric is used. In addition, since the figure which does not become point symmetry is used, even if it rotates 180 degree | times, it does not become the same figure.
The dimension of the square frame of the marker is a predetermined value (for example, the length “L” of one side of the square).

FIG. 3 is an example of the display screen 106 of the mobile terminal device.
FIG. 3A illustrates a screen that displays a captured image.
The captured image 391 of the screen 106 illustrated includes a captured image of the component 14 and a captured image of the marker 16.
The marker 16 of the photographed image 391 is arranged in a predetermined positional relationship with respect to the component 14. The predetermined positional relationship is, for example, a state where the marker 16 is placed parallel to the front surface of the component 14 and then tilted from the front to the back and placed horizontally.
Here, the reason for using the front of the component 14 as a reference is that the front of the component is associated with the front coordinate information of the three-dimensional CAD data of the component database, and the imaging condition calculated from the shape of the marker image of the captured image is three-dimensional. This is to make it applicable to CAD data. (Details will be described later) The contractor knows where the front of each component 14 is.

FIG. 3B illustrates a component information screen.
The illustrated order screen is configured by associating a part photo, a part number, an inventory button, and an order button.
The stock button is a button for receiving an instruction for requesting display of stock information such as the number of stocks. The order button is a button used when inputting order information of the part.

Here, the three coordinate systems handled by the component management system 1 will be described with reference to FIG.
Three coordinate systems parts management system 1 is handled, three-dimensional space coordinate system _{(X m, Y m, Z} m) and an electronic camera coordinate system _{(X c, Y c, Z} c) and photographed image plane coordinate system (X _p , Y _p ).
The three-dimensional space coordinate system (X _m , Y _m , Z _m ) is a three-dimensional space with the marker as the origin. In the three-dimensional space coordinate system, parts and the electronic camera of the mobile terminal device 300 are arranged.
The electronic camera coordinate system (X _c , Y _c , Z _c ) is a three-dimensional space with the electronic camera of the mobile terminal device 300 as the origin.
The captured image plane coordinate system (X _p , Y _p ) is a two-dimensional space that represents a two-dimensional image captured by the electronic camera of the mobile terminal device 300.

FIG. 5 is a diagram for explaining the component database.
The component database 193 includes a CAD table 1931 and a component information table 1932 for management.
The CAD table 1931 is tabular data in which a part number (primary key) 19311 is associated with the three-dimensional CAD data 19312 of the part specified by the part number.
The part number 19311 is a unique character string.
The three-dimensional CAD data 19312 is numerical data of a three-dimensional spatial coordinate system (Xm, Ym, Zm) that represents the three-dimensional shape of the part.
The component information table 1932 is tabular data in which a component number (main key) 19321 is associated with component data 19322 of a component specified by the component number. The part number 19321 is the same as the part number 19311. The part data 19322 is data including a part inventory quantity and a part number.
Accordingly, the three-dimensional CAD data 19312 and the part data 19322 are associated with each other by a part number.

（４―１−４）《形状正規化処理》部品情報サーバー装置１００は、マーカー候補画像の４本の線分の４つの交点座標（＝４つの頂点座標）を用いて、マーカー候補画像を、正方形にアフィン変換処理して、正方形マーカー候補画像に変換する。
（４−１−５）《画素数正規化処理》部品情報サーバー装置１００は、正方形マーカー候補画像に対して、標準マーカーの画素数と一致するように拡大／縮小処理して、正規化マーカー候補画像に変換する。
（４−１−６）《マーカー認識処理》部品情報サーバー装置１００は、相互相関関数（たとえば、類似度を算出できるということが数学的にすでに分かっている下記の数式３）を用いて、正規化マーカー候補画像と標準マーカー画像（二次元画像）の類似度を算出して、最も類似度の高い正規化マーカー候補画像をマーカーとして認識して、検出する。

なお、ｔは、標準マーカー画像である。
Ｍは、標準マーカー画像の横方向の画素数である。
Ｎは、標準マーカー画像の縦方向の画素数である。
ｆは、正規化マーカー候補画像である。
ｋは、正規化マーカー候補画像ｆまたは標準マーカー画像ｔの横方向の画素位置である。
ｌは、正規化マーカー候補画像ｆまたは標準マーカー画像ｔの縦方向の画素位置である。
ｆ［ｋ］［ｌ］は、正規化マーカー候補画像ｆの画素位置（ｋ、ｌ）の画素値（０または１）である。
ｔ［ｋ］［ｌ］は、標準マーカー画像ｔの画素位置（ｋ、ｌ）の画素値（０または１）である。
オーバーライン付ｆは、正規化マーカー候補画像ｆの画素値の平均値である。
オーバーライン付ｔは、標準マーカー画像ｔの画素値の平均値である。 Next, a detailed processing procedure of (4) << Calculation of shooting conditions >> will be described with reference to FIG.
(4) << Calculation of shooting conditions >>
(4-1) << Marker extraction >>
A marker image is cut out from the captured image by the following six procedures.
(4-1-1) << Binarization Processing of Captured Image >> The component information server apparatus 100 uses the brightness information (for example, arithmetic average of RGB values) of the captured image, and uses an appropriate threshold (for example, brightness). Binarization processing is performed using a half of the maximum value of information) to generate a binarized image.
Note that the brightness information of the captured image uses, for example, a gray scale value when the captured image is gray scale. In addition, when the captured image has RGB values, it may be calculated by an arithmetic average or a weighted average of RGB values.
(4-1-2) << Contour Part Pixel Extraction Process >> The product information server apparatus 100 scans the black pixel region of the binarized image from the end, and uses a contour extraction algorithm using four connected pixels to detect black pixels. And the extracted black pixel is used as a contour pixel.
Here, the contour extraction algorithm using four connected pixels is a scan of a binarized image from the end, with the detected untracked black pixel as the starting contour pixel, and the four connected pixels centered on this pixel as an inverse. The black pixel is detected by tracking clockwise and is used as a contour pixel, and the black pixel is detected by 4-connected pixel tracking with this pixel as the next center, and is repeated until the start point is returned. The contour extraction algorithm may use 8-connected pixels.
(4-1-3) << Rectangular Image Cutting Process >> The component information server apparatus 100 calculates a line segment by performing a polygonal line approximation process on the contour pixel, and is surrounded by four calculated line segments. The square image extracted is used as a marker candidate image.
The polygonal line approximation process is, for example, using the Hough transform formula (Formula 1 below) mathematically known that a straight line can be detected from an array of a large number of contour pixels, according to the following procedure. A straight line passing through the pixels may be calculated.
(A) A Hough transform formula (Formula 1 below) is applied to the contour pixel to convert it into a sine curve.
(B) A sine curve is associated with the pixel.
(C) Extract coordinates (ρ ₀ , θ ₀ ) of points where a predetermined number or more of sine curves intersect.
(D) Substituting the coordinates (ρ ₀ , θ ₀ ) of the extracted point into the Hough transform equation (the following equation 1), and passing through the pixel corresponding to the sine curve intersecting this point (the following equation 2) Is calculated.

(4-1-4) << Shape Normalization Process >> The component information server apparatus 100 uses the four intersection coordinates (= four vertex coordinates) of the four line segments of the marker candidate image to convert the marker candidate image into An affine transformation process into a square is performed to convert it into a square marker candidate image.
(4-1-5) << Pixel Number Normalization Process >> The component information server apparatus 100 enlarges / reduces the square marker candidate image so as to match the number of pixels of the standard marker, thereby normalizing marker candidates. Convert to image.
(4-1-6) << Marker Recognition Processing >> The component information server apparatus 100 uses the cross-correlation function (for example, the following mathematical expression 3 that has already been mathematically known to be able to calculate the similarity) The similarity between the normalized marker candidate image and the standard marker image (two-dimensional image) is calculated, and the normalized marker candidate image with the highest similarity is recognized and detected as a marker.

Note that t is a standard marker image.
M is the number of pixels in the horizontal direction of the standard marker image.
N is the number of pixels in the vertical direction of the standard marker image.
f is a normalized marker candidate image.
k is the horizontal pixel position of the normalized marker candidate image f or the standard marker image t.
l is the vertical pixel position of the normalized marker candidate image f or the standard marker image t.
f [k] [l] is the pixel value (0 or 1) at the pixel position (k, l) of the normalized marker candidate image f.
t [k] [l] is a pixel value (0 or 1) at the pixel position (k, l) of the standard marker image t.
The overlined f is an average value of the pixel values of the normalized marker candidate image f.
The overlined t is an average value of the pixel values of the standard marker image t.

For example, by using arDetectMarker of ARToolKit which is an augmented reality programming library, a marker can be easily extracted from a captured image. ARToolKit is an open source programming library developed and provided by the Human Interface Technology Laboratory at the University of Washington.

Ｘ_pは、撮影画像平面座標系における画像のＸ座標値である。Ｙ_pは、撮影画像平面座標系における画像のＹ座標値である。Ｐ_ijは、所与のキャリブレーションの行列要素である。ここで、ｉ，ｊ＝１、２，３である。ｈは、カメラ原点と画像平面の距離である。
（４−２−２）《座標変換行列の算出》
部品情報サーバー装置１００は、座標変換行列関係式（たとえば、座標変換行列関係式として、すでに数学的に分かっている下記の数式５）に対して、カメラ座標系の撮影画像のマーカーの４つの頂点座標値［Ｘ_ck 、Ｙ_ck 、Ｚ_ck ］（ｋ＝１、２，３、４）と、三次元空間座標系の標準マーカーの４つの頂点座標値［Ｘ_mk、Ｙ_mk 、Ｚ_mk ］（ｋ＝１、２，３、４）とを代入して、１２個の式を作成してこれらを解いて、座標変換行列の回転移動成分の９つの行列要素Ｒ_ij （ｉ，ｊ＝１、２，３）と、座標変換行列の平行移動成分の３つの行列要素Ｔ_i （ｉ＝１、２，３）と、を算出して、電子カメラ座標系（Ｘ_c 、Ｙ_c 、Ｚ_c ）と三次元空間座標系（Ｘ_m、Ｙ_m 、Ｚ_m ）との座標変換行列関係式（下記の数式６）を算出する。

Ｘ_c は、カメラ座標系における画像のＸ座標値である。Ｙ_c は、カメラ座標系における画像のＹ座標値である。Ｚ_c は、カメラ座標系における画像のＺ座標値である。
Ｘ_mは、三次元空間座標系における画像のＸ座標値である。Ｙ_m は、三次元空間座標系における画像のＹ座標値である。Ｚ_m は、三次元空間座標系における画像のＺ座標値である。
Ｒ_ij は、座標変換行列の回転移動成分の行列要素である。
Ｔ_i は、座標変換行列の平行移動成分の行列要素である。
ここで、ｉ，ｊ＝１、２，３である。

Ｒ_a 、Ｒ_b 、・・、Ｒ_i は、算出した回転移動成分の行列要素である。Ｔ_a 、Ｔ_b 、Ｔ_c は、算出した平行移動成分の行列要素である。
なお、たとえば、前述したＡＲＴｏｏｌＫｉｔのarGetTransMat を用いることにより、撮影画像のマーカーを用いた座標変換行列関係式を、容易に求めることができる。 (4-2) << Shooting Condition Calculation Process >> The component information server apparatus 100 calculates a coordinate transformation matrix as a shooting condition from the shape of the marker image detected from the photographed image by the following two procedures.
(4-2-1) << Coordinate transformation of photographed image >>
The component information server apparatus 100 uses the camera calibration matrix relational expression (Formula 4) calculated by the well-known camera calibration process to calculate the coordinate value of the captured image plane coordinate system (X _p , Y _p ) of the captured image. Conversion into coordinate values of the camera coordinate system (X _c , Y _c , Z _c ).

X _p is the X coordinate value of the image in the captured image plane coordinate system. Y _p is the Y coordinate value of the image in the captured image plane coordinate system. P _ij is a matrix element of a given calibration. Here, i, j = 1, 2, and 3. h is the distance between the camera origin and the image plane.
(4-2-2) << Calculation of coordinate transformation matrix >>
The component information server device 100 uses the four vertexes of the marker of the captured image in the camera coordinate system with respect to the coordinate transformation matrix relational expression (for example, the following mathematical expression 5 already mathematically known as the coordinate transformation matrix relational expression). The coordinate values [X _ck , Y _ck , Z _ck ] (k = 1, 2, 3, 4) and the four vertex coordinate values [X _mk , Y _mk , Z _mk ] ( Substituting k = 1, 2, 3, 4), twelve equations are created and solved, and nine matrix elements R _ij (i, j = 1, 2, 3) and three matrix elements T _i (i = 1, 2, 3) of the translation component of the coordinate transformation matrix, and the electronic camera coordinate system (X _c , Y _c , Z _c ) And a coordinate transformation matrix relational expression (Formula 6 below) between the three-dimensional space coordinate system (X _m , Y _m , Z _m ).

X _c is the X coordinate value of the image in the camera coordinate system. Y _c is the Y coordinate value of the image in the camera coordinate system. Z _c is the Z coordinate value of the image in the camera coordinate system.
X _m is the X coordinate value of the image in the three-dimensional spatial coordinate system. Y _m is the Y coordinate value of the image in the three-dimensional spatial coordinate system. Z _m is the Z coordinate value of the image in the three-dimensional spatial coordinate system.
R _ij is a matrix element of the rotational movement component of the coordinate transformation matrix.
T _i is a matrix element of the translation component of the coordinate transformation matrix.
Here, i, j = 1, 2, and 3.

R _a , R _b ,..., R _i are matrix elements of the calculated rotational movement components. T _a , T _b and T _c are matrix elements of the calculated translation component.
For example, by using arGetTransMat of ARTToolKit described above, a coordinate transformation matrix relational expression using a marker of a captured image can be easily obtained.

The detailed processing procedure of (5) << part data extraction >> will be described with reference to FIGS.
FIG. 8 is a diagram for explaining the detailed processing procedure of (5) << part data extraction >>.
(5) << Part data extraction >>
(5-1) << Cutout Processing of Component Photographed Image >> The component information server apparatus 100 removes a marker image whose position is known from the background image from the photographed image, and binarizes this image to perform binary processing. A binarized image is generated, an outermost quadrangle of black pixels of the binarized image is created, and a component captured image is cut out from the captured image using the outermost quadrangle.
In addition, what is necessary is just to use the picked-up image which image | photographed the screen of the color different from the color of components, for example, leaning behind the components for the picked-up image. Since the background image of the photographed image is a pixel of screen color, the background image can be removed from the photographed image by removing it.
FIG. 11 shows an example of a component photographed image 3911 cut out by the component photographed image clipping process.

Returning to FIG. 8, the detailed processing procedure of (5) << Part data extraction >> will be continued.
(5-2) << Part CG Image Generation Processing >> The part information server apparatus 100 reads the three-dimensional CAD data 19312 and the part number from the part database 193 and uses the CAD data 19312 as a calculated coordinate transformation matrix relational expression (= Substituting into [X _m , Y _m , Z _m ] of the relational expression calculated using Equation 5 to convert to three-dimensional CAD data [X _c , Y _c , Z _c ] of the camera coordinate system, The rendered three-dimensional CAD data is rendered to generate a part CG image, and the read part number is associated with the part CG image.
FIG. 11 shows an example of component CG image generation processing. In the illustrated component CG image generation process, first, the calculated coordinate transformation matrix relational expression 195 performs a matrix operation on the three-dimensional CAD data 19312 substituted for [X _m , Y _m , Z _m ], and the result Is output to [X _c , Y _c , Z _c ] as three-dimensional CAD data 19313 converted.
Next, in the illustrated part CG image generation process, the input of the converted three-dimensional CAD data 19313 is received, a rendering process is performed, and a part CG image 19314 is output.

Returning to FIG. 8, the detailed processing procedure of (5) << Part data extraction >> will be continued.
(5-3) << Part Image Recognition Process >>
The component information server device 100 calls the image recognition program, passes the extracted component photographed image data 3911 and the generated component CG image data 19314 to the image recognition program, and receives the similarity calculated by the image recognition program. The part number associated with the part CG image having a similarity equal to or higher than a predetermined threshold is held as the part number of the part similar to the part image.
Note that the image recognition program calculates the feature amount data of the received component photographed image and component CG image, compares them, and calculates the similarity. As the image recognition program, for example, the technique of Non-Patent Document 1 may be used.
FIG. 11 shows an example of image recognition processing. In the illustrated image recognition process, the image recognition program 183 inputs the component photographed image 3911 and the component CG image 1934 and outputs the similarity 196.

Returning to FIG. 8, the detailed processing procedure of (5) << part data extraction >> will be continued.
(5-4) << Part Data Search Processing >> The part information server apparatus 100 searches the part information table 1932 of the part database 193 using the held part number 19311 as a search key, and matches the search key with the part number that matches the search key. The component data 19322 associated with 19321 is read out.

FIG. 9 is a detailed configuration diagram of the mobile terminal device 300.
The mobile terminal device 300 includes a CPU 301, a display unit 302, an operation unit 303, a camera unit 304, a network communication unit 308, a storage unit 309, and a dedicated program.
The CPU 301, the display unit 302, the input unit 303, the camera unit 304, the communication unit 308, and the storage unit 309 are connected by BUS199.

The CPU 301 is a central processing unit.
The display unit 302 is a liquid crystal display device or an organic EL display device.
The operation unit 303 is a key button. The operation unit 302 may be a soft key button displayed on the display unit 103.
The camera unit 304 is an electronic camera that generates digital image data by reading an electrical signal from an imaging device that records an exposure target. The electronic camera 304 shoots the parts on the construction site and the marker 16 to generate photographic image data 391 including the part image and the marker image.
The microphone 305 is a device for inputting sound.
The speaker 306 is a device that outputs sound.
The power supply unit 307 is a secondary battery.
The communication unit 308 is a device that communicates radio signals with the base station.

The storage unit 309 is a semiconductor memory or a magnetic memory.
The storage unit 309 includes a captured image storage area 3091 and a replacement part candidate storage area 3092, and stores the operating system 385 and a dedicated program.
The captured image storage area 3091 stores captured image data 391.
The component information screen storage area 3092 stores component information screen data 197.

The operating system 385 manages and manages the hardware of the mobile terminal device 300 (for example, the CPU 301, the display unit 302, the input unit 303, the camera unit 304, the network communication unit 308, the storage unit 309, the BUS399, etc.). It is basic software that controls and provides a service that can use these hardware for application software (for example, a dedicated program).

In addition to this, a captured image transmission unit 310 and a component information screen reception display unit 320 are provided. Each of these means is realized by each dedicated program, and functions when the dedicated program is interpreted and executed by the CPU 301.

The captured image transmission unit 310 transmits captured image data 391 including the component image and the marker image generated by the electronic camera 304 to the component management server apparatus 100 via the communication unit 308 and the base station.
The component information screen reception display means 320 receives the component information screen data 197 returned from the component management server apparatus 100 via the base station and the communication unit 308 and displays it on the display unit 302.

FIG. 10 is a detailed configuration diagram of the component management server apparatus 100.
The component management server apparatus 100 includes a CPU 101, a network communication unit 104, a storage unit 109, and a dedicated program.
The CPU 101, the network communication unit 104, and the storage unit 109 are connected by BUS199.

The CPU 101 is a central processing unit.
The network communication unit 104 is a LAN adapter.

The storage unit 109 is a semiconductor memory or a magnetic memory.
The storage unit 109 includes a standard marker storage area 091, a parts information screen storage area 092, a parts information storage area 093, a captured image storage area 094, and a coordinate transformation matrix storage area 095, and a database program 180. The image recognition program 183, the operating system 185, and a dedicated program are stored.

The standard marker storage area 091 stores standard marker information 191 including dimension information.
The component information screen storage area 092 stores component information screen data 197.
The component storage area 093 stores a component database 193.
The component database 193 is a database having component data and three-dimensional CAD data, and is managed by the database program 180.
The captured image storage area 094 stores captured image data 391.
The coordinate transformation matrix storage area 095 stores coordinate transformation matrix relational expression data 195.

The image recognition program 183 is a program using a known image recognition algorithm.
The operating system 185 manages and controls computer hardware (for example, the CPU 101, the network communication unit 104, the storage unit 109, and the BUS 199), and applies application software (for example, the database program 180 and a dedicated program). On the other hand, it is basic software that provides services that can use these hardware.

In addition, the marker detection means 120, the imaging condition calculation means 130, the part CG image generation means 140, the part data extraction means 150, the part information screen creation means 160, the captured image reception means 170, the part information screen. Replying means 175. Each of these means is realized by each dedicated program and functions by the CPU 101 interpreting and executing the dedicated program.

The captured image reception 170 receives captured image data 391 including a component image and a marker image transmitted from the mobile terminal device 300 via the network communication unit 104.

The marker detection unit 120 detects a marker image from the received captured image data 391.
The details of the processing of the marker detection unit 120 are described in the section of (4-1) << Marker extraction >> in the description of the detailed processing procedure of (4) << Calculation of shooting conditions >> in FIG.

The imaging condition calculation means 130 uses the four vertex coordinates of the detected marker image and the four vertex coordinates of the standard marker 191 to use the coordinates of the electronic camera coordinate system (X _c , Y _c , Z _c ) and three-dimensional. A coordinate transformation matrix relational expression 195 with the coordinates of the spatial coordinate system (X _m , Y _m , Z _m ) is calculated.
Details of the processing of the photographing condition calculation means 130 are described in the section of (4-2) << Shooting condition calculation processing >> in the description of the detailed processing procedure of (4) << Calculation of photographing conditions >> in FIG.

The component CG image generation unit 140 reads the three-dimensional CAD data 19312 from the CAD table 1931 of the component database 193, and uses the coordinate transformation matrix relational expression 195 calculated by the imaging condition calculation unit 130 for the three-dimensional CAD data 19312. By applying and rendering, a part CG image (= two-dimensional digital image data) of the electronic camera coordinate system (X _c , Y _c , Z _c ) is generated and associated with the part number 19311.

The component specifying unit 155 compares the generated component CG image data with the component image cut out from the captured image data, recognizes the image, and corresponds to the component CG image data image-recognized with a similarity equal to or higher than a predetermined threshold. The assigned part number 19311 is specified.

The component specifying unit 155 includes a component photographed image cutout function, a replacement component image recognition function, and a search function.
Details of the processing of the part photographed image cut-out function will be described in the section of (5-2) << Part photographed image cut-out process >> in the detailed processing procedure of (5) << Extraction of part data >> in FIG. It was.
Details of the processing of the replacement part image recognition function are described in the section of (5-3) << Part Image Recognition Process >> in the description of the detailed processing procedure of (5) << Extraction of Part Data >> in FIG.

The component data extraction unit 150 searches the component database 193 using the identified component number 19311 as a search key, and reads the component data 19322.
Details of the processing of the part data extracting means 150 are described in the section of (5-4) << Parts Data Retrieval Process >> in the detailed processing procedure of (5) << Parts Data Extraction >> in FIG.

The component information screen creation means 160 creates the component information screen data 197 using the read component data.

The component information screen reply unit 175 returns the created component information screen data 197 to the mobile terminal device 300.

DESCRIPTION OF SYMBOLS 1 Parts management system 16 Marker 100 Parts management server apparatus 130 Imaging condition calculation means 140 Parts CG image generation means 150 Parts data extraction means 160 Parts information screen creation means 170 Photographed image reception means 175 Parts information screen return means 193 Parts database 1931 CAD table 19311 Part number 1912 Three-dimensional CAD data 1932 Part information table 19321 Part number 19322 Part data 195 Coordinate transformation matrix relational data 197 Part information screen data 300 Portable terminal device 310 Photographed image transmission means 320 Part information screen reception display means 391 Photographed image data

Claims (4)

A replacement part management method using a portable terminal apparatus with an electronic camera and a part management apparatus having an image recognition program,
In a replacement part management method using a part database for managing standard marker information, three-dimensional CAD data and part data associated with a part number, provided in the part management apparatus,
A component photographing step of photographing a component and generating photographed image data together with a marker arranged in a predetermined positional relationship with respect to the component by an electronic camera of the mobile terminal device;
A captured image transmission step of transmitting the captured image data to the parts management device;
A marker detection step in which the component management device detects a marker image from the received captured image data;
An imaging condition calculation step in which the component management device calculates a coordinate transformation matrix using the detected marker image and the shape information of the standard marker,
The component management apparatus applies the calculated coordinate transformation matrix to the three-dimensional CAD data read from the component database, performs rendering processing, generates a component CG image, and associates the component number with the component CG image generation step. When,
The component management apparatus calculates the similarity by comparing the generated component CG image data and the component image of the photographed image data using the image recognition program, and the component number associated with the component CG image having a high similarity A component identification step for extracting
A parts information screen creation reply step in which the parts management device creates part information screen data using the part data of the extracted part number and sends it back to the mobile terminal device;
A replacement part management method, characterized in that it is performed by a procedure including: Standard marker storage area for storing standard marker information;
Parts information screen storage area for storing parts information screen data,
Three-dimensional CAD data associated with a part number; a part storage area for storing a part database for managing part data associated with a part number;
A captured image storage area for storing captured image data including a component image and a marker image;
Storage means having
A photographed image receiving means for receiving photographed image data obtained by photographing a component together with a marker arranged in a predetermined positional relationship with respect to the component;
Marker detection means for detecting a marker image from received photographed image data;
An imaging condition calculation means for calculating a coordinate transformation matrix using the detected marker image and dimension information of the standard marker,
A component CG image generation unit that generates a component CG image by applying the coordinate transformation matrix calculated by the imaging condition calculation unit to the three-dimensional CAD data read from the component database, generates a component CG image, and associates it with the component number When,
A component that specifies a component number associated with a high-similarity component CG image by calculating the similarity between the generated component CG image data and the component image of the captured image data with respect to a component database having an image recognition program Specific means,
A part data extraction means for searching by using the specified part number as a search key and receiving the extracted part data;
Part information screen creation means for creating part information screen data using the extracted part data;
A component information screen return means for returning the created component information screen data to the mobile terminal device;
A component management apparatus comprising: A computer program that causes a computer to operate as the component management apparatus according to claim 2 by being incorporated in the computer. The computer-readable recording medium which recorded the computer program of Claim 3.

Images