JP5510924B2 - Bread identification device and program - Google Patents

Description

  The present invention relates to bread identification, and more particularly to an apparatus for identifying a bread type by image recognition and a program thereof.

  There is a request to identify the type of bread. For example, there is a request for automatically recognizing how many types of bread are baked from a bread-baking kettle and managing production. In addition, there is a demand for automatically recognizing the type of bread and the number of each type of bread on the tray when selling bread, and automating price calculation and management of the number of sales. It is troublesome to manually input the type and number of breads near the bread-baking pot, and it is not preferable to press the keys with hands covered with flour. Skills are necessary for the salesperson to accurately identify the type of bread, and it is impossible for an unskilled salesperson to input manually from a POS terminal.

  Regarding the identification of the type of food, Patent Document 1 (Japanese Patent Laid-Open No. 2000-225049A) discloses image recognition of the type of sushi on a plate. In patent document 1, when the similar pattern exists with the same sales price, it proposes changing the arrangement to a plate for every kind of sushi. However, this method is unlikely to be effective for identifying the type of bread.

The problems in bread identification are as follows.
1) The color of the bread is basically dark brown, and the shape is almost circular.
2) There is a large difference in shape among individuals even in the same type of bread. For example, even in the same type of croissant, the shape of the bread varies widely, such as the aspect ratio of the bread and the presence or absence of lateral protrusions in the center of the bread.
3) Some of the basic features are common, such as fried bread and not a perfect circle in plan view, such as piroshki and curry bread.
4) There are very similar breads such as strained and breaded bread.

JP2000-225049A

  An object of the present invention is to accurately identify the type of bread by image recognition.

The bread identifying apparatus of the present invention is an apparatus for identifying a type of bread by recognizing a color image of a pan from a camera,
A cutout unit that cuts out three types of color images from the color image of the pan, the color image of the inner region of the pan, the color image of the intermediate region, and the color image of the outer region ;
A contour data extraction unit for obtaining a feature value related to the outline of the bread from the color image of the bread, and a feature value related to the texture of the bread from the color image of the bread using the luminance component in the color data of the bread. Texture data extraction unit of
An inner color data extraction unit for obtaining a feature amount in the color space of the color image of the inner area of the pan;
An outer color data extraction unit for obtaining a feature amount in the color space of the color image of the outer region of the pan;
An intermediate color data extraction unit for obtaining a feature amount in the color space of the color image of the intermediate area of the pan;
An identification unit for identifying the type of bread based on the feature amount obtained by each of the extraction units.

The bread identification program of the present invention is
In order to identify the type of pan by recognizing the pan color image from the camera,
Computer
A cutout unit that cuts out three types of color images from the color image of the pan, the color image of the inner region of the pan, the color image of the intermediate region, and the color image of the outer region ;
An outline data extraction unit for obtaining a feature amount related to the outline of the bread from the color image of the bread,
A texture data extraction unit for obtaining a feature amount related to the texture of the bread from the color image of the bread using a luminance component in the color data of the bread ;
An inner color data extraction unit for obtaining a feature amount in the color space of the color image of the inner area of the pan;
An outer color data extraction unit for obtaining a feature amount in the color space of the color image of the outer region of the pan;
An intermediate color data extraction unit for obtaining a feature amount in the color space of the color image of the intermediate area of the pan;
It functions as an identification unit that identifies the type of bread based on the feature value obtained by each extraction unit.

Preferably, the identification unit is configured to identify a bread type among all types of bread, and at least a first type identification unit that cannot identify a bread type. And a second-stage identifying unit for identifying the bread type for the bread type candidate identified in the first stage .

In the present invention, the type of bread is identified by using the features of the inner, middle, and outer color spaces when the bread is divided into the inner, middle, and outer sides . The bread outline is effective for identifying round bread such as Anpan and roll bread, elongated bread such as Danish and French bread, square bread of bread and pizza triangle. The texture can be used to identify the texture of the bread crust itself, such as roll bread and pie bread, the texture of the bread crumbs of the deep-fried bread, the sugar grains in the bread sprinkled with sugar, and the beans in the bean bread. The color data on the inside of the pan reflects the topping color, the color data on the outside reflects the color of the pan itself, and the baked color of the pan is different between the intermediate region and the outer region. All of these features are insufficient to identify bread alone, but by combining these features, many types of bread can be identified almost accurately.

  However, it is difficult to distinguish very similar breads such as strained bread and grained bread. Therefore, if the bread type is substantially identified by the first stage identification and then identified in the similar bread group by the second stage identification, the bread type can be accurately identified. The identification at the second stage may be performed only for those that are difficult to identify at the first stage, for example.

  If the type of bread is accurately identified, the price calculation can be automated when the bread is sold, and the sales quantity can be automatically determined for each type of bread. In addition, the production quantity and type of bread can be determined accurately.

Block diagram of identification device of embodiment The flowchart which shows the feature-value extraction (pre-processing) algorithm in an Example Diagram showing bread cut from background (tray) The figure which divides | segments bread into 3 area | regions of the inner side of the left side, the middle part of the center, and the outer side of the right side The flowchart which shows the learning algorithm in the execution example The flowchart which shows the identification algorithm in an Example The figure which shows the relationship between the number of features in the 1st step | paragraph identification, and a correct answer rate It is a figure which shows the example of identification in the 2nd stage, The bread | pan on the right side is "Milk France", and the bread | pan on the left side is "Almond France".

  In the following, an optimum embodiment for carrying out the present invention will be shown. The scope of the present invention should be determined according to the understanding of those skilled in the art based on the description of the scope of the claims, taking into account the description of the specification and well-known techniques in this field.

1 to 8 show an embodiment. FIG. 1 shows an identification device 2 according to the embodiment, 4 is an appropriate light source, 6 is a CCD camera, and another camera may be used. A background separation unit 8 separates the pan and the background from the color image of the CCD camera 6. The background is, for example, a tray on which a plurality of breads are placed, and the bread is cut out from the trays based on a change in hue or a change in hue and a change in luminance. If the pans are placed in contact with each other, the images of the pans are separated from each other by starting with the contour at the part where the pan is not in contact and extending to the part where the pan is in contact. When the breads are in contact with each other, the brightness changes suddenly at the contact portion. The cutout unit 10 divides the bread image cut out by the background separation unit 8 into, for example, three regions on the inside (region with topping), the middle, and the outside .

  The light source 4 and the CCD camera 6 may not be provided in the identification device 2 and these may be separated from the identification device 2. The color space to be used is RGB, H (hue) S (saturation) V (luminance), etc., and luminance and lightness are used in the same meaning. Although separation into clusters is used for image recognition, a recognition method that does not use clusters may be used.

  Data from the background separation unit 8 is sent to the contour data extraction unit 12, and features such as area, circularity, average and variance of distance from the center of gravity to the contour, aspect ratio, filling degree, and contour Hu moment are extracted. . The circularity is, for example, a ratio of the square of the bread area and the bread contour length. The aspect ratio is the ratio of the maximum bread length to the length in a direction perpendicular thereto, and the degree of filling is the ratio of the bread area to the smallest rectangular area that can accommodate the bread. For example, a second moment is used as the Hu moment, and the Hu moment is well known in image recognition. Hu moments do not change with image rotation, reduction, enlargement, etc., and if they are the same kind of pan, it can be expected that the Hu moments are similar to each other.

  The texture data extraction unit 14 extracts texture data from the entire pan image, and obtains a co-occurrence matrix as texture data here, but obtains a high-frequency component in a two-dimensional Fourier transform or a two-dimensional autocorrelation function. Also good. In the co-occurrence matrix, i and j are subscripts representing pixels, and the pixel value is luminance. Assuming that the pixel i, j is relatively separated by δ = (d, θ), and the pixel value of the pixel i is Li, the co-occurrence matrix indicates the probability that the pixel value of the pixel j is Lj. ing. The co-occurrence matrix changes every δ, but since it is a digital image, the value of d is 0, 1, 2, etc., and the value of θ is limited to 0 °, 45 °, 90 °, 135 °, etc. The number of matrices is also limited.

  The inner color data extraction unit 16, the intermediate color data extraction unit 18, and the outer color data extraction unit 20 extract feature amounts in the color space with respect to the three regions of the inside, the middle, and the outside cut out from the pan. The feature amount in the color space includes an average value and dispersion of color coordinates in a color space such as RGB or HSV, and a Hu moment of luminance. The inner color data extraction unit 16 extracts feature amounts from the color data of the inner region, the intermediate color data extraction unit 18 extracts feature amounts from the color data of the intermediate region, and the outer color data extraction unit 20 extracts the feature amount of the outer region. Extract features from color data. In the embodiment, 91 types of feature quantities are extracted from the pan color image in this way. However, when it is determined which feature quantity is to be used by learning, the required feature quantities are reduced to, for example, about 10 types, and therefore the extraction units 12 to 20 extract only the necessary feature quantities.

  The learning unit 22 learns what kind of feature value combination to perform image recognition, the input is a correct answer regarding the type of bread, the output is the feature value combination, or the feature value combination and its weight, etc. To the units 24 and 26. Image recognition is, for example, to decompose a learning sample (features extracted from color images of various types of bread) into, for example, clusters for each type of bread. If one cluster contains only one type of bread, basically the type of bread can be accurately identified. If one cluster includes a plurality of types of bread, or if the separation between the clusters is incomplete, the type of bread may be erroneously recognized. Therefore, first-stage image recognition and second-stage image recognition are performed.

  The first-stage image recognition unit 24 recognizes the type of bread using the feature amount specified by the learning unit 22. The first-stage image recognizing unit 24 identifies the type of bread from among all types of bread that are targeted. For identification, for example, the center coordinates of clusters (clusters) of the same kind of bread in the space determined by the feature value and the cluster radius are used, and the type is determined from which cluster the feature value of the input bread belongs. . The output is the top 3 candidates for bread types, and if one bread type can be reliably identified, the top 3 candidates are all set to the same bread type. For example, this case corresponds to a case where it certainly belongs to a cluster consisting of only one kind of bread. When there is a possibility of belonging to a plurality of clusters, or when belonging to a cluster including a plurality of types of bread, such as strained and breaded bread, a plurality of types of bread are output as the top three candidates. If the top three candidates match, the bread type is output assuming that correct recognition has been performed. If there is a mismatch among the top three candidates, the second-stage image recognition unit 26 identifies that it is difficult to identify. Even when the top three candidates match, the second-stage identification may be performed.

  The second-stage image recognition unit 26 differs from the recognition unit 24 in the combination of feature amounts used, and the types of pans that are identification candidates are limited to the cluster specified by the first-stage identification and the surrounding clusters. For example, in a cluster composed of strained bread and grained bread, either is identified, and the range for identifying the type of bread is separated. When the top three candidates obtained by the second stage recognition unit 26 match each other, it is output that the correct answer has been obtained. By repeating this operation for each individual bread, the type and number of breads on the tray can be determined. If the top three candidates of the second stage recognition unit 26 do not match, the operator inputs a correct answer from the learning unit 22 or the like. The learning unit 22 performs the learning again when the frequency of incorrect answers is high or when a type of bread is added.

  Reference numeral 30 denotes a storage medium that stores the program of the identification device 2. For example, by storing the program in the storage medium 30 in a personal computer that can control the light source 4 and the CCD camera 6, the identification device 2 of the embodiment can be obtained.

FIG. 2 shows the preprocessing algorithm, and FIGS. 3 and 4 show the contents of the preprocessing. The bread and the background are separated by changing the hue or the like (step 1). Thus, only the bread portion can be extracted from the bread image on the left tray in FIG. Divide the pan into three areas: inner, middle, and outer due to changes in HSV . In FIG. 4, the central topping region can be separated because it has a different color tone from the middle pan region. Since the middle region and the outer region have different colors, they can also be separated. The intermediate area and the outer area reflect the baking color of bread, and the inner area reflects the topping as described above (step 2). Extract feature values of the outline of the entire pan (step 3), and extract feature values in the color space such as average, variance, and Hu moment of HSV values for each of the inner, middle, and outer regions (step 4). . Then, texture features such as a co-occurrence matrix are extracted for the entire pan (step 5).

  FIG. 5 shows a learning algorithm, and the first-stage learning searches for an optimum feature amount for the purpose of discriminating all kinds of bread from each other (step 11). A new bread that has not been used for the feature amount search is used, and is identified by the obtained feature amount. Here, if the top three candidates are equal and match the actual type, the answer is correct, and if they do not match, the answer is incorrect. When the accuracy rate is low, the feature amount to be used is changed, and the search for the feature amount is continued until a desired accuracy rate (for example, 80% or more) is obtained (step 12).

  The second stage learning is aimed at distinguishing similar breads from each other, and the feature amount used is a grain ann if black sesame is on the anpan, and an annunciator to the end of the bread if there is an eggplant grain. If there is, there is almond france, and if there is no cut to the end, it is milk france (step 13). This feature reflects the bakery's professional tradition, such as using black sesame for Anpan and eggplant for Koshi An. In step 14, a feature quantity for identifying similar breads is searched for. In step 15, a new bread that has not been used for the feature quantity search is identified. If the accuracy rate is low, the feature quantity is changed.

  FIG. 6 shows an identification algorithm. In the learning stage, the feature quantities in the first stage and the second stage are found, and the feature quantities to be used and their weights are input to the recognition units 24 and 26. The type of bread is identified according to the feature amount (step 21), and if the reliability at the first stage identification is high, the type of bread is output as it is. If the reliability is low, the type of bread is output by the second stage identification (steps 22 to 24).

  As a result, since the type of bread on the tray and the number of each type can be obtained at the store, the price can be automatically determined, and the sales quantity can be managed at the same time. Therefore, even if there are many types of bread, bread can be sold by an unfamiliar clerk. In the case of bread production management, the type and number of baked bread can be automatically obtained in the same manner.

  FIG. 7 shows the relationship between the number of feature values used for first-stage identification and the accuracy rate. When 29 types of bread are purchased 6 pieces each and 91 types of feature quantities are extracted, the correct answer rate is obtained, and the correct answer rate is the probability of identifying the bread that was not used for learning. The correct answer rate became the highest when the feature amount was about 10 to 20, and the correct answer rate did not improve even if the number of features was further increased. FIG. 7 shows the accuracy rate in the first-stage identification. When the second-stage identification is performed, the accuracy rate is almost 100%.

  FIG. 8 shows an example in the second stage identification, where the bread on the right is milk France and the bread on the left is almond France. The actual distinction is whether there is a cut to the end of the bread, which has nothing to do with the use of milk cream or almond cream to season the bread. In the first stage, the cluster of milk France and almond France is identified, and in the second stage, these are identified in the cluster.

In the embodiment, the following effects can be obtained.
1) The type and number of bread can be recognized automatically.
2) By dividing the bread into an inner area, an intermediate area and an outer area , the topping characteristics and the bread baking characteristics can be used separately.
3) Increase the recognition rate by using the features of the pan outline. Round pans, square pans, triangular pans, pans with many contours, etc. can be identified by contour features.
4) Increase the recognition rate by using the texture features of bread. For example, curry bread and anpan do not differ greatly in the shape of the bread itself. However, curry bread has a crumb texture on its surface, which is characteristic of fried bread. Also, pie bread such as croissants and other breads have different textures.
5) By combining the identification of the first level and the identification of the second level, the type of bread can be accurately identified.

2 Identification Device 4 Light Source 6 CCD Camera 8 Background Separation Unit 10 Cutout Unit 12 Outline Data Extraction Unit 14 Texture Data Extraction Unit 16 Inner Color Data Extraction Unit 18 Intermediate Color Data Extraction Unit 20 Outer Color Data Extraction Unit 22 Learning Unit 24 First Stage Image recognition unit 26 Second stage image recognition unit 30 Storage medium

Claims (2)

A device for recognizing pan types by recognizing pan color images from a camera,
A cutout unit that cuts out three types of color images from the color image of the pan, the color image of the inner region of the pan, the color image of the intermediate region, and the color image of the outer region ;
An outline data extraction unit for obtaining a feature amount related to the outline of the bread from the color image of the bread,
A texture data extraction unit for obtaining a feature amount related to the texture of the bread from the color image of the bread using a luminance component in the color data of the bread ;
An inner color data extraction unit for obtaining a feature amount in the color space of the color image of the inner area of the pan;
An outer color data extraction unit for obtaining a feature amount in the color space of the color image of the outer region of the pan;
An intermediate color data extraction unit for obtaining a feature amount in the color space of the color image of the intermediate area of the pan;
An identification unit that identifies a type of bread based on a feature amount obtained by each of the extraction units. In order to identify the type of pan by recognizing the pan color image from the camera,
Computer
A cutout unit that cuts out three types of color images from the color image of the pan, the color image of the inner region of the pan, the color image of the intermediate region, and the color image of the outer region ;
An outline data extraction unit for obtaining a feature amount related to the outline of the bread from the color image of the bread,
A texture data extraction unit for obtaining a feature amount related to the texture of the bread from the color image of the bread using a luminance component in the color data of the bread ;
An inner color data extraction unit for obtaining a feature amount in the color space of the color image of the inner area of the pan;
An outer color data extraction unit for obtaining a feature amount in the color space of the color image of the outer region of the pan;
An intermediate color data extraction unit for obtaining a feature amount in the color space of the color image of the intermediate area of the pan;
A bread identification program that functions as an identification unit that identifies the type of bread based on the feature amount obtained by each of the extraction units.