JP2024083601A - Detection support device, learning device, detection support method, learning method, and program - Google Patents

Abstract

[Problem] A processing process based on human vision for a content image is executed as processing on a processor. [Solution] A detection support device that supports the detection of the presence or absence of an unexpected pattern due to the continuity of a pattern in a content image generated by repeatedly arranging a predetermined pattern includes a content image acquisition unit that acquires the content image, a gaze feature image generation unit that uses a gaze feature learning model to estimate gaze features that are characteristics of the gaze when detecting the pattern from the content image and generates a gaze feature image showing the estimated gaze features for each pixel in the content image, and an analysis result output unit that displays the gaze feature image, and the gaze feature learning model is a trained model that has learned the correspondence between the content image for learning and the gaze features of a person who detected the pattern from the content image for learning. [Selected Figure] Figure 6

Description

The present invention relates to a detection support device, a detection support method, and a program that help people to more easily detect defects in content images.

In the field of building materials, design has long been considered an important added value, and decorative sheets with designs such as wood grain or abstract patterns are used by adhering them to the interior and exterior of buildings, furniture, and other furnishings. Some designs for such decorative sheets are created by repeating a certain pattern as a unit, resulting in a certain pattern being synchronized.

In designs where such predetermined patterns are synchronized (hereafter referred to as content images or visual content), the continuity of the patterns can create unexpected patterns or shadows, resulting in defects that impair the design. Such defects cannot be detected at the stage of designing a single pattern, and are often only detected when an image is created in which a single pattern is repeatedly arranged, and this image is observed from a certain distance away. This is thought to be because a person observing an image visually senses some kind of spatial regularity (pattern) in the content image where the pattern is repeated.

In general, there are differences in the visual features that trained humans (experts) and untrained humans (non-experts) can detect in content images of the same design. This is thought to be because the visual features that humans can detect are greatly influenced not only by the physical characteristics of the content image, but also by the visual characteristics of the human observer.

In other words, while an expert is able to detect defects in the appearance of such content images, an unskilled person is often unable to detect such defects. This is thought to be because an expert has learned how to detect defects in content images through training. In other words, an expert is thought to have established a unique way of looking at content images and a unique method of processing them during the visual information processing process. If such unique ways of looking at content images could be quantified, even an unskilled person would be able to detect defects in such content images without undergoing burdensome training.

L. Itti, C. Koch, E. Niebr: "A model of saliency-based visual attention for rapid scene analysis", IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 20, Issue: 11: P. 1254-1259, November 1998. D. Gao, V. Mahadevan, N. Vasconcelos: "On the plausibility of the discriminant center-surround hypothesis for visual salience", Journal of Vision, Vol. 8,13,June 2008. J. Harel, C. Koch, P. Perona: “Graph-Based Visual Salience”, Advances in neural information processing systems, 19:545-552, January, 2006.

By reproducing on a processor a process that mimics the basic processing method of human vision, it is possible to reproduce to a certain extent the way humans see content images as processing performed by a device. For example, human vision receives light information from the eye and extracts two-dimensional information indicating brightness and color on the retina. Human vision then transmits the extracted information to the visual cortex of the brain. In the visual cortex of the brain, the intensity of brightness, spatial discontinuity (edges), continuity (gradients), chromaticity representation of color information (representation of the three primary colors red, green, and blue, and opponent color representations such as red-green/yellow-blue), and other factors are processed individually based on the brightness and other information obtained from vision.

Furthermore, in the visual cortex of the brain, the spatial contrast (between the center and periphery) and directional continuity/discontinuity of the processed information are processed, and then combinations of these are processed, and so on. By repeating further processing using the results of the processing, increasingly higher-level and more complex patterns are processed in stages. If each of these processes could be realized sequentially on a processor, it would be possible to process the same type of information as human vision, and it would be possible to control certain circuits (processing) to act strongly and others to act weakly.

On the other hand, for example, one could focus on the process by which an experienced person with a good eye (an expert) looks at an object and notices a defect, and imagine replicating that observation process on a processor. In other words, if the knowledge and intuition that people acquire through a certain amount of training, known as experiential knowledge, could be reproduced on a processor, even an unskilled person would be able to use the results of the processor's processing to achieve the same results as an expert.

The present invention has been made in consideration of the above circumstances, and its purpose is to provide a detection support device, a learning device, a detection support method, a learning method, and a program that can execute the human visual processing process for content images as processing on a processor.

In order to solve the above-mentioned problems, a detection support device according to one aspect of the present invention is a detection support device that supports the detection of the presence or absence of unexpected patterns that arise in a content image generated by repeatedly arranging a predetermined pattern, due to the continuity of the pattern in the content image, and includes a content image acquisition unit that acquires the content image, a gaze feature image generation unit that uses a gaze feature learning model to estimate gaze features that are characteristics of the gaze when detecting the pattern from the content image and generates a gaze feature image showing the estimated gaze features for each pixel in the content image, and an analysis result output unit that displays the gaze feature image, and the gaze feature learning model is a trained model that has learned the correspondence between the content image for learning and the gaze features of a person who detected the pattern from the content image for learning.

In order to solve the above-mentioned problems, a learning device according to one aspect of the present invention is a learning device that generates a gaze feature learning model used by a detection support device that supports detection of the presence or absence of unexpected patterns due to the continuity of patterns in a content image generated by repeatedly arranging a predetermined pattern, the gaze feature learning model being a trained model that has learned the correspondence between the content image for learning and gaze features that are characteristics of the gaze of a person who detected the pattern from the content image for learning, and is configured to be able to select either deep learning or transfer learning using an existing learning model as a method for generating the gaze feature learning model, and includes a deep learning unit that generates the gaze feature learning model using the selected method.

In order to solve the above-mentioned problems, a detection support method that is one aspect of the present invention is a detection support method performed by a computer that is a detection support device that supports the detection of the presence or absence of unexpected patterns that arise in a content image generated by repeatedly arranging a predetermined pattern due to the continuity of the pattern in the content image, in which a content image acquisition unit acquires the content image, a gaze feature image generation unit estimates gaze features that are characteristics of the gaze when detecting the pattern from the content image using a gaze feature learning model, generates a gaze feature image that shows the estimated gaze features for each pixel in the content image, an analysis result output unit displays the gaze feature image, and the gaze feature learning model is a learned model that has learned the correspondence between the content image for learning and the gaze features of a person who detected the pattern from the content image for learning.

In order to solve the above-mentioned problems, a learning method that is one aspect of the present invention is a learning method performed by a computer that is a learning device that generates a gaze feature learning model used by a detection support device that supports the detection of the presence or absence of unexpected patterns due to the continuity of patterns in a content image generated by repeatedly arranging a predetermined pattern, the gaze feature learning model is a trained model that has learned the correspondence between the content image for learning and gaze features that are characteristics of the gaze of a person who detected the pattern from the content image for learning, and a deep learning unit is configured to be able to select either deep learning or transfer learning using an existing learning model as a method for generating the gaze feature learning model, and generates the gaze feature learning model using the selected method.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned detection assistance device.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned learning device.

As described above, according to the present invention, the processing of content images based on human vision can be executed as processing on a processor.

1 is a block diagram showing an example of a configuration of a detection support device 100 according to a first embodiment of the present invention. 4 is a flowchart showing a flow of processing performed by the detection support device 100 of the first embodiment of the present invention. FIG. 2 is a diagram showing an example of an image to be processed in the first embodiment of the present invention. 4 is an example of a visual feature image according to the first embodiment of the present invention. FIG. 11 is a diagram showing an analysis result of the first embodiment of the present invention. FIG. 11 is a diagram showing an analysis result of the first embodiment of the present invention. FIG. 11 is a diagram showing an analysis result of the first embodiment of the present invention. FIG. 11 is a block diagram showing an example of a configuration of a detection support device 100A according to a second embodiment of the present invention. 10 is a flowchart showing a flow of processing performed by a detection support device 100A according to a second embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of a learning device 200 according to the embodiment. 1 is a flowchart showing a flow of processing performed by the learning device 200 of the embodiment. 4 is a flowchart showing an example of the operation of a process performed by the learning device 200 according to the present embodiment. FIG. 13 is a block diagram showing an example of a configuration of a detection support device 100B according to a third embodiment of the present invention. 13 is a flowchart showing a flow of processing performed by a detection support device 100B according to a third embodiment of the present invention.

The detection support device according to the embodiment will be described below with reference to the drawings.

First Embodiment
First, the first embodiment will be described.
The ability of humans to make certain judgments or feel something is wrong based on information obtained visually is dependent on the processing of the visual nerve mechanism in the human brain. Currently, the relatively early stages of brain processing are becoming clear. Therefore, the detection support device 100 is considered as a device that performs processing that models this processing. By having the detection support device 100 perform processing of the visual nerve mechanism, it becomes possible to more accurately reproduce visual information processing in the human brain.

The detection support device 100 of this embodiment uses, for example, a reference image and an inspection image as processing targets. The reference image is an image in which a defect has been detected by an expert. The inspection image is an image in which the defect has been eliminated through processing by the expert.

By targeting a reference image and a test image, the detection assistance device 100 can capture the difference in how an expert and an unexpert see both images as a difference in content features that model the processing process of the neural mechanisms of human visual perception. In other words, the detection assistance device 100 clearly presents the difference in how both images appear for each of several indicators, thereby showing what properties of the test image differ from the reference image and to what extent, thereby helping even an unexpert to easily detect defects.

Note that, in the following, an example will be described in which the image to be processed (content image) is a still image, but this is not limited to this. The content image may also be a moving image, video, etc.

FIG. 1 is a block diagram showing an example of the configuration of a detection support device 100 according to a first embodiment of the present invention. The detection support device 100 includes, for example, a content image selection unit 101, a visual feature selection unit 102, a visual feature image generation unit 103, an image feature selection unit 104, a content feature amount calculation unit 105, an analysis method selection unit 106, an analysis unit 107, a content image DB (database) 108, a visual feature DB 109, a visual feature image storage unit 110, an image feature DB 111, a content feature amount storage unit 112, an analysis method DB 113, an analysis result storage unit 114, and an analysis result output unit 115. The content image selection unit 101 is an example of a "content acquisition unit".

The content image selection unit 101 acquires a content image. A content image is an image that expresses a design that is generated by repeatedly arranging a predetermined pattern. For example, a content image is an image that shows the design of a decorative sheet such as wallpaper that is used as a building material.

The content image selection unit 101 acquires an image selected by a user or the like from among a plurality of content images stored in the content image DB 108 as a content image. The method of selection by the user or the like may be any method. For example, the content image selection unit 101 refers to the content image DB 108 and causes the content image to be displayed on a display unit (not shown). The content image selection unit 101 acquires an image selected by the user or the like operating an external input device such as a mouse or keyboard as a content image.

The content images are not limited to those stored in the content image DB 108, but may be images acquired by the detection assistance device 100 via any input means, such as a portable memory, a scanner, or a communication network.

The visual feature selection unit 102 selects visual features.
Visual features are features that can be recognized by vision at a relatively early stage of human brain processing, such as luminance, chromaticity, contrast, gradient, edges, optical flow, and the like.
The visual features include luminance, chromaticity, red-green chromaticity, yellow-blue chromaticity, orientation, luminance gradient, chromaticity gradient, red-green gradient, yellow-blue gradient, orientation gradient, luminance contrast, chromaticity contrast, red-green contrast, yellow-blue contrast, orientation contrast, and the like.
The visual feature may be an index representing the ease with which a person's eye is attracted. Examples of the index representing the ease with which a person's eye is attracted include a visual attention model, a gaze prediction model, a saliency model, and a saliency model. For example, the methods described in Non-Patent Document 1, Non-Patent Document 2, Non-Patent Document 3, and the like can be used.
The visual feature selection unit 102 selects a visual feature selected by a user or the like from a plurality of visual features stored in the visual feature DB 109, for example.

The visual feature image generating unit 103 generates a visual feature image by applying visual features to the content image. The visual feature image is an image that shows the visual features in the content image, and is, for example, an image in which the degree of visual feature (visual feature amount) calculated for each pixel in the content image corresponds to the position coordinates of the pixel. The content image used here is the image selected by the content image selecting unit 101. The visual features used here are the visual features selected by the visual feature selecting unit 102. The visual feature image generating unit 103 stores the generated visual feature image in the visual feature image storage unit 110.

The image feature selection unit 104 selects image features. Image features are image features extracted using known image processing techniques, such as texture features that can extract spatial regularity (patterns) in a design formed by arranging multiple identical designs. Examples of texture features include contrast, correlation, angular second moment, and uniformity.

The image feature selection unit 104 selects, for example, an image feature selected by a selection operation by a user or the like from among a plurality of image features stored in the image feature DB 111. The image feature selection unit 104 outputs the acquired image feature to the content feature amount calculation unit 105.

The content feature calculation unit 105 calculates the content feature. The content feature is an index indicating the degree of the feature related to the appearance of the content image, and is, for example, a statistical quantity that statistically indicates the image features in the visual feature image, calculated by applying image features to the visual feature image.

For example, if brightness is selected as the visual feature and contrast is selected as the image feature, the content feature amount is a value that indicates what kind of contrast is formed by the brightness state that can be recognized by human vision in the content image. The visual feature image used here is an image generated by the visual feature image generation unit 103. The image feature used here is an image feature selected by the image feature selection unit 104.

The content feature amount calculation unit 105 outputs the calculated content feature amount to the analysis unit 107. The content feature amount calculation unit 105 also stores the calculated content feature amount in the content feature amount storage unit 112.

The content feature amount may be the degree of visual features (visual feature amount). In this case, the content feature amount calculation unit 105 calculates the content feature amount, for example, by using a visual feature image (without using image features).

The analysis method selection unit 106 selects an analysis method. The analysis method is a method for presenting content features, and is, for example, information indicating the type of graph showing the content features. Graph types include, for example, line graphs, bar graphs, pie charts, radar charts, etc.

The analysis method selection unit 106 selects, for example, an image feature selected by a selection operation by a user or the like from among a plurality of image features stored in the analysis method DB 113. The image feature selection unit 104 outputs the acquired image feature to the content feature amount calculation unit 105.

The analysis unit 107 applies an analysis method to the content features to generate information for presenting the content features in the content image. The analysis unit 107 stores the generated information in the analysis result storage unit 114. The analysis unit 107 also outputs the generated information to the analysis result output unit 115.

The content image DB 108 stores content images. For example, the content image DB 108 stores a content image associated with identification information that uniquely identifies the content image. The content image is acquired by the detection assistance device 100 via any input means, such as an external input device such as a mouse or keyboard, or a portable memory, a scanner, or a communication network, and is stored in the content image DB 108.

In the content image DB 108, a reference image and an inspection image obtained by processing the reference image may be stored in association with each other, or the content images may be stored in a state of being classified according to the type of content image. The type of content image may be classified, for example, according to the method of combining patterns, or whether or not there are irregularities on the sheet surface when it is made into a decorative sheet.

The visual feature DB 109 stores visual features. For example, the visual feature DB 109 stores perceptual features associated with identification information that uniquely indicates the visual feature. The perceptual features are acquired by the detection support device 100 via, for example, an external input device or an input means, and are stored in the visual feature DB 109.

The visual feature image storage unit 110 stores the visual feature image generated by the visual feature image generation unit 103. The visual feature image storage unit 110 stores, for example, a visual feature image associated with identification information that uniquely indicates the visual feature image, identification information of the content image used to generate the visual feature image, and identification information of the visual feature used to generate the visual feature image.

Image feature DB111 stores image features. For example, image features associated with identification information that uniquely indicates the image feature are stored in image feature DB111. The image features are acquired by the detection support device 100 via, for example, an external input device or an input means, and are stored in image feature DB111.

The content feature storage unit 112 stores the content feature calculated by the content feature calculation unit 105. The content feature storage unit 112 stores, for example, a content feature associated with identification information that uniquely identifies the content feature, identification information of a visual feature image used to calculate the content feature, and identification information of an image feature used to calculate the content feature.

The analysis method DB113 stores analysis methods. For example, the analysis method DB113 stores an analysis method associated with identification information that uniquely indicates the analysis method. The analysis method is acquired by the detection support device 100 via an external input device or an input means, for example, and is stored in the analysis method DB113.

The analysis result storage unit 114 stores the analysis results (information for presenting content features) by the analysis unit 107. The analysis result storage unit 114 stores, for example, the analysis results associated with identification information that uniquely indicates the analysis results, the identification information of the content features used in the analysis, and the analysis method used in the analysis.

The analysis result output unit 115 outputs the analysis result (information for presenting content features) by the analysis unit 107. The analysis result output unit 115 outputs the analysis result to a display unit (not shown), for example, and causes the analysis result to be displayed on the display unit.

FIG. 2 is a flowchart showing a flow of processing performed by the detection support device 100 according to the first embodiment of the present invention.
Step S10:
The detection support device 100 selects a content image by the content image selection unit 101. The content image selection unit 101 selects a content image by referring to the content image DB 108, and outputs the selected content image to the visual feature image generation unit 103.
Step S11:
The detection support device 100 selects a visual feature by the visual feature selection unit 102. The visual feature selection unit 102 selects a visual feature by referring to a visual feature DB 109, and outputs the selected visual feature to the visual feature image generation unit 103.
Step S12:
The detection support device 100 generates a visual feature image by the visual feature image generation unit 103. The visual feature image generation unit 103 generates a visual feature image by calculating the perceptual features selected in step S11 in the content image selected in step S10. The visual feature image generation unit 103 outputs the generated visual feature image to the content feature amount calculation unit 105.

Step S13:
The detection support device 100 selects an image feature by the image feature selection unit 104. The image feature selection unit 104 selects an image feature by referring to the image feature DB 111, and outputs the selected image feature to the content feature amount calculation unit 105.
Step S14:
The detection support device 100 calculates a content feature amount in the content image by the content feature amount calculation unit 105. The content feature amount calculation unit 105 calculates the content feature amount by calculating the image feature selected in step S13 in the visual feature image generated in step S12.
Step S15:
The detection support device 100 selects an analysis method by the analysis method selection unit 106. The analysis method selection unit 106 selects an analysis method by referring to the analysis method DB 113, and outputs the selected analysis method to the analysis unit 107.
Step S16:
The detection support device 100 performs analysis (generation of information for presenting feature quantities of a content image) by the analysis unit 107. The analysis unit 107 generates information indicating the feature quantities of the content image calculated in step S14 by the analysis method selected in step S15. The analysis unit 107 outputs the generated information to the analysis result storage unit 114 and the analysis result output unit 115.

Step S17:
The detection support device 100 stores the analysis results (information for presenting the feature amounts of the content image) obtained in step S16 in the analysis result storage unit 114.
Step S18:
The detection support device 100 outputs the analysis result (information for presenting the feature amount of the content image) analyzed in step S16 by the analysis result output unit 115 to a display unit (not shown) or the like.

3 is a diagram showing an example of a reference image and an inspection image according to the first embodiment of the present invention, in which the reference image is shown on the left and the inspection image is shown on the right.
As shown in Figure 3, for example, the reference image and the test image look so similar that they seem to be the same design. There seems to be almost no difference between the reference image and the test image. However, an expert can detect defects in the reference image and process the reference image to look like the test image based on the detected defects.
In this embodiment, the detection support device 100 can visualize and present the difference in appearance between the reference image and the test image due to human visual perception as a difference in content features. In other words, the detection support device 100 can numerically indicate the degree to which the characteristics of the test image differ from the reference image. In this way, the detection support device 100 can help an unskilled person to recognize the difference between the two images, which at first glance appear to be almost the same.

Figure 4 is a diagram showing an example of a visual feature image in the first embodiment of the present invention. Figure 4 shows an example in which contrast is selected as the visual feature.

In Figure 4, an example of a visual feature image in a reference image is shown on the left, and an example of a visual feature image in a test image is shown on the right, with three visual feature images shown for each scale for the reference image and test image. The scale is an index that indicates the size of the space (image size) in which the visual feature (in this example, contrast) is calculated, with finer scale values (smaller image size for calculating contrast) shown upwards and coarser scale values (larger image size) shown downwards. The three visual feature images, 1x1, indicate the visual feature image when one reference image is connected vertically and horizontally (one image), 2x2, indicate the visual feature image when two images are connected vertically and horizontally (four images), and 3x3, indicate the visual feature image when three images are connected vertically and horizontally (nine images).

That is, the top row of FIG. 4 shows the visual feature images of the reference image and the test image with image sizes of (1×1), (2×2), and (3×3) from the left when the scale value is calculated as 8 (denoted as "scl8"). Similarly, the top row of FIG. 4 shows the visual feature images of the reference image and the test image with image sizes of (1×1), (2×2), and (3×3) from the left when the scale value is calculated as 16 (denoted as "scl16"). The top row of FIG. 4 shows the visual feature images of the reference image and the test image with image sizes of (1×1), (2×2), and (3×3) from the left when the scale value is calculated as 8 (denoted as "scl8").

As shown in Figure 4, when comparing the visual feature image generated from the reference image and the visual feature image generated from the test image, for example, when comparing the two images corresponding to (1 x 1) on scale 8 and the visual feature image corresponding to (1 x 1) on scale 16, there appears to be a difference compared to when comparing the reference image and test image in Figure 3.

When differences are found in the visual feature images, it indicates that there is a difference in the visual features (in this example, contrast) between the reference image and the test image. In other words, by converting both images into visual feature images, it is possible to emphasize the difference in appearance caused by the visual features in both images.

Figure 5A is a diagram showing the analysis results of the first embodiment of the present invention. Figure 5A shows an example in which a line graph is selected as the analysis method.

Figure 5A shows three line graphs for each scale value. Each line graph shows the correlation difference of the visual feature images in the reference image and the test image in four directions. The four directions are 0 [deg], 45 [deg], 90 [deg], and 135 [deg] from a predetermined reference axis set in the image. From the left, the three line graphs show the correlation difference for image sizes of (1x1), (2x2), and (3x3).

FIG. 5B shows a (2×2) correlation difference at scale 8 out of the multiple correlation differences shown in FIG. 5A.
As shown in FIG. 5B, for example, in a (2×2) scale of 8, the correlation differences in the directions of 0 degrees and 90 degrees tend to exhibit larger values than the correlation differences in other directions.

In areas where relatively large differences in visual features are shown, it is shown that there are relatively large differences between the two images. In other words, by converting both images into visual feature images and showing the visual features, it is possible to quantitatively show the differences caused by the visual features in both images.

Figure 5C is a diagram showing the analysis results of the first embodiment of the present invention. Figure 5C shows an example in which a radar chart is selected as the analysis method.

Figure 5C shows three radar charts for each scale value. Each radar chart shows the content features (in this example, the contrast of the visual feature image) in the reference image and the test image in eight directions. The eight directions are up, down, left, right, and top right, top left, bottom left, and bottom right from the center of the image. From the left, the three radar charts show the correlation difference for image sizes of (1x1), (2x2), and (3x3).

As shown in FIG. 5C, for example, the contrast in all directions of the reference image at (1×1) scale 32 tends to be greater than the contrast of the test image.

Where relatively large differences in content features are shown, it is shown that there are relatively large differences between the two images. In other words, by showing the content features of both images, it is possible to quantitatively show the differences in appearance between the two images.

As described above, the detection support device 100 of the first embodiment includes a content image selection unit 101 (an example of a "content image acquisition unit") and a content feature amount calculation unit 105. The content image selection unit 101 acquires information about the content image. The content feature amount calculation unit 105 calculates the content feature amount by applying visual features to the content image. As a result, according to the detection support device 100 of the first embodiment, visual features that are features that can be recognized by human vision can be applied to the content image, and therefore a processing process based on human vision can be executed as processing on a processor.

The detection support device 100 of the first embodiment may further include a visual feature image generation unit 103. The visual feature image generation unit 103 generates a visual feature image. The visual feature image is an image in which visual features, which are the degree of visual features for each pixel in a content image, are associated with the pixels. As a result, the detection support device 100 of the first embodiment can show visual features associated with pixels of the content image, and can clearly show which parts of the content image have what visual features.

In addition, in the detection support device 100 of the first embodiment, the visual feature image generation unit 103 generates a visual feature image using at least one selected from luminance, chromaticity, contrast, edge, optical flow, and skewness as a visual feature. As a result, according to the detection support device 100 of the first embodiment, it is possible to generate a visual feature image with higher accuracy by using a technique similar to the visual appearance among existing image processing techniques.

In addition, in the detection support device 100 of the first embodiment, the visual feature image generation unit 103 generates a visual feature image by using a recognition index that indicates the ease with which the human eye can recognize a content image as a visual feature. As a result, the detection support device 100 of the first embodiment can generate a visual feature image that is closer to the recognition by the human eye.

In addition, in the detection support device 100 of the first embodiment, the recognition index includes at least one of a visual attention model, a gaze prediction model, a saliency model, and a saliency model. As a result, the detection support device 100 of the first embodiment can generate visual feature images with higher accuracy using existing models.

In addition, in the detection support device 100 of the first embodiment, the content feature amount calculation unit 105 calculates the content feature amount by applying image features to the visual feature image. As a result, according to the detection support device 100 of the first embodiment, the visual feature image can be subjected to image processing techniques to statistically process the degree of the features indicated by the visual features, and the content feature amount can be indicated more quantitatively.

As our understanding of human brain functions progresses, many features that can be recognized visually are being discovered, and these features may also be included in visual features.

Second Embodiment
The second embodiment will be described below with reference to the drawings. The detection support device 100A of this embodiment differs from the above-mentioned embodiment in that it presents a pseudo line of sight when an expert detects a defect in a content image. The detection support device 100A presents the line of sight of an expert, thereby supporting the detection of defects, so that even an unskilled person can easily detect defects. In this embodiment, only the configuration different from the first embodiment will be described, and the same reference numerals will be used for the configuration similar to that of FIG. 1 according to the first embodiment, and the description thereof will be omitted unless particularly necessary.

FIG. 6 is a block diagram showing an example configuration of a detection support device 100A according to the second embodiment. The detection support device 100A includes, for example, a gaze feature learning model selection unit 116, a gaze feature image generation unit 117, a gaze feature learning model DB 118, and a content feature amount calculation unit 105A. The detection support device 100A does not include a visual feature selection unit 102 and a visual feature image generation unit 103.

In the following, in this embodiment, the content image is described as a still image, but similar to the first embodiment, this may be applied to other content images such as moving images and video.

The gaze feature learning model selection unit 116 selects a gaze feature learning model. The gaze feature learning model is a model that estimates gaze features in a content image, generated by a machine learning technique. The gaze feature is information that indicates the features related to the gaze when an expert detects a defect in a content image, and is, for example, a gaze feature image described below, or information that statistically indicates the degree to which each area of the content image is visible.

The gaze feature learning model is generated, for example, by performing machine learning using learning data in which each of a number of different learning content images (learning content images) is associated with the gaze feature records of an expert who viewed each learning content image.

The means for acquiring gaze features may be a commercially available dedicated gaze measurement device (for example, Tobii's Tobii Pro Glasse 2, which is a body-worn measurement device, or the Tobii Pro X2, X3, which is a stationary measurement device), or a calculation method that combines a consumer camera with a gaze estimation method may be used. Gaze features are generally stored in chronological order as the coordinates of the viewpoint measured at a fixed sampling time, and these are converted into coordinates on the image for use.

The machine learning method used here may be any method, but is performed, for example, using an estimation model such as a deep neural network. A deep neural network, for example, has an input layer, an output layer, and a configuration in which the layers are connected by multiple convolutional layers and pooling layers in between. When a learning content image is input to the input layer of the multilayer neural network, learning is repeated so that the information output from the output layer of the multilayer neural network becomes the gaze feature associated with the learning content image, and the coupling coefficients and bias values that couple each layer are determined. A gaze feature learning model is generated by determining the coupling coefficients and bias values of the estimation model.

The gaze feature learning model selection unit 116 selects, for example, a gaze feature learning model selected by a selection operation by a user or the like from among a plurality of gaze feature learning models stored in the gaze feature learning model DB 118. The gaze feature learning model selection unit 116 outputs the acquired gaze feature learning model to the gaze feature image generation unit 117.

The gaze feature image generating unit 117 generates a gaze feature image by applying the gaze features estimated by the gaze feature learning model to the content image. The gaze feature image is an image that indicates the degree of gaze features in the content image. As a method for obtaining the gaze feature image from the gaze features, for example, a method is used in which the accumulation of viewpoints on the image within the measurement time is approximated as a probability distribution and a heat map is output.

A gaze feature image is an image in which gaze features are condensed into information in a single image. Here, one method of condensing information is, for example, to count the gaze distribution for each pixel in the content image to create a two-dimensional histogram, and then to approximate the peaks of the histogram using a two-dimensional normal distribution and express the intensity as real values in the range [0, 1] (commonly called a heat map).

The content image used by the gaze feature image generation unit 117 is an image selected by the content image selection unit 101. The gaze feature used by the gaze feature image generation unit 117 is estimated by inputting the content image selected by the content image selection unit 101 into the gaze feature learning model selected by the gaze feature learning model selection unit 116. The gaze feature image generation unit 117 outputs the generated gaze feature image to the content feature amount calculation unit 105A.

The gaze feature learning model DB118 stores gaze feature learning models. The gaze feature learning model DB118 stores, for example, gaze feature learning models associated with identification information that uniquely indicates the gaze feature learning model. The gaze feature learning model is generated, for example, by an external learning server, acquired by the detection support device 100 via an external input device or input means, and stored in the analysis method DB113. The gaze feature learning model DB118 may store models according to the type of content image. This makes it possible to select a model according to the type even if the expert's way of looking at the content image differs depending on the type of content image, making it possible to estimate gaze features with greater accuracy.

The content feature amount calculation unit 105A calculates the content feature amount by applying image features to the gaze feature image. The content feature amount in this embodiment is, for example, a statistical amount that statistically indicates the image features in the gaze feature image.

For example, when contrast is selected as an image feature, the content feature amount is a value indicating what kind of contrast the expert's gaze forms in the content image. The gaze feature image used here is an image generated by the gaze feature image generation unit 117. The image feature used here is an image feature selected by the image feature selection unit 104.

The content feature amount may be the degree of gaze feature, in which case the content feature amount is the gaze feature image itself.

Figure 7 is a flowchart showing an example of the processing performed by the detection support device 100A according to this embodiment. The processing shown in steps S23 and S25 to S28 in Figure 7 is similar to the processing shown in steps S13 and S15 to S18 in Figure 2, and therefore a description thereof will be omitted.

Step S20:
The detection support device 100A outputs the content image selected by the content image selection unit 101 to the gaze characteristic image generation unit 117.
Step S21:
The detection support device 100A selects a gaze feature learning model by the gaze feature learning model selection unit 116. The gaze feature learning model selection unit 116 selects a gaze feature learning model by referring to the gaze feature learning model DB 118, and outputs the selected gaze feature learning model to the gaze feature image generation unit 117.
Step S22:
Detection support device 100A generates a gaze feature image by gaze feature image generation unit 117. Gaze feature image generation unit 117 generates a gaze feature image by estimating gaze features in the content image selected in step S20 using the gaze feature learning model selected in step S21. Gaze feature image generation unit 117 outputs the generated gaze feature image to content feature amount calculation unit 105A.
Step S24:
The detection support device 100A calculates a content feature amount in the content image by the content feature amount calculation unit 105A. The content feature amount calculation unit 105A calculates the content feature amount by calculating the image feature selected in step S23 in the gaze feature image generated in step S22.

As described above, in the detection support device 100A of the second embodiment, the content feature amount calculation unit 105A calculates the content feature amount by applying, to the content image, gaze features that indicate the characteristics of the gaze of the person viewing the content image. As a result, the detection support device 100A of the second embodiment achieves the same effects as those described above.

The detection support device 100A of the second embodiment further includes a gaze feature image generation unit 117. The gaze feature image generation unit 117 generates a gaze feature image in which the gaze feature of each pixel in the content image corresponds to the position coordinates of the pixel. The content feature amount calculation unit 105A calculates the content feature amount using the gaze feature image. As a result, the detection support device 100A of the second embodiment achieves the same effects as those described above.

In addition, in the detection support device 100A of the second embodiment, the gaze feature is estimated using a gaze feature learning model generated by performing machine learning using learning data that associates a content image with the performance of the gaze feature in the content image. As a result, the detection support device 100A of the second embodiment makes it possible to more accurately estimate the gaze feature of an expert in a content image based on past performance.

8 is a block diagram showing an example of the configuration of a learning device 200 according to an embodiment. The learning device 200 is a device that generates a gaze feature learning model.
The learning device 200 includes, for example, a learning content image acquisition unit 201, a gaze information acquisition unit 202, a learning gaze feature image generation unit 203, a deep learning unit 204, a learning content image DB 205, a gaze information memory unit 206, a gaze feature image memory unit 207, and a gaze feature learning model DB 208.

The learning content image acquisition unit 201 acquires learning content images. The learning content images are learning data used when performing machine learning on an estimation model, and are information to be input (set) to the input layer of the estimation model.

The learning content image acquisition unit 201 acquires a set of learning content images, which is a set of images selected by a user or the like according to the amount of learning from among a plurality of learning content images stored in the learning content image DB 205. The method of selection by the user or the like may be any method. The learning content image acquisition unit 201 outputs the acquired learning content images to the learning gaze characteristic image generation unit 203.

The gaze information acquisition unit 202 acquires gaze information (gaze features). The gaze information is information about the gaze of an expert on a learning content image, and is, for example, information indicating time-series changes in the gaze of an expert viewing a learning content image. The gaze information acquisition unit 202 selects gaze information corresponding to the learning content image from, for example, a plurality of visual features stored in the gaze information storage unit 206. The gaze information acquisition unit 202 outputs the acquired gaze information to the learning gaze feature image generation unit 203.

The learning gaze characteristic image generation unit 203 generates a learning gaze characteristic image by applying gaze information to the learning content image. The method of generating the learning gaze characteristic image is similar to the method of generating the gaze characteristic image by the gaze characteristic image generation unit 117, and therefore a description thereof will be omitted. The learning gaze characteristic image generation unit 203 outputs the generated learning gaze characteristic image to the deep learning unit 204 and also stores it in the gaze characteristic image storage unit 207.

The deep learning unit 204 generates a gaze feature learning model by performing learning (deep learning) using the learning gaze feature image as learning data. The deep learning unit 204 stores the generated gaze feature learning model in the gaze feature learning model DB 208.

9 is a flowchart showing an example of the operation of the process performed by the learning device 200 according to the present embodiment. In FIG. 9, the flow of the operation of the process of generating a gaze feature learning model by deep learning is shown.
Step S50:
Study device 200 acquires study content images by study content image acquisition unit 201. It is preferable for study content image acquisition unit 201 to acquire a large number of study content images. This is because, in general, a good learning effect can be obtained by preparing a large number of study data (study content images) with a wide variety of images.
Step S51:
Study device 200 acquires gaze information corresponding to a learning content image by gaze information acquisition unit 202.
Step S52:
The learning device 200 generates a learning gaze characteristic image by applying the gaze information acquired in step S51 to the learning content image acquired in step S50 using the learning gaze characteristic image generation unit 203.
Step S53:
Learning device 200 determines whether or not learning gaze characteristic images have been generated for all of the learning content images acquired in step S50. If learning device 200 has generated learning gaze characteristic images for all of the learning content images, it executes the process shown in step S54. If learning device 200 has not generated learning gaze characteristic images for all of the learning content images, it returns to the process shown in step S51.
Step S54:
The learning device 200 generates a gaze feature learning model by performing deep learning using the learning gaze feature image as learning data.

Fig. 10 is a flowchart showing an example of the operation of the process performed by the learning device 200 according to the present embodiment. Fig. 10 shows the flow of the operation of the process of generating a new learning model by performing transfer learning on a gaze feature learning model by deep learning (hereinafter, also simply referred to as a learning model).
Step S60:
The learning device 200 configures an input layer and an output layer of an estimation model by the deep learning unit 204. The estimation model is a deep learning model with a multi-layer structure of intermediate layers (pooling layer and convolution layer). Information on each pixel in the learning content image is input to the input layer. The output layer is a fully connected layer that performs normalization. This output layer is configured to output a decimal value between "1" and "0".
Step S61:
The deep learning unit 204 determines whether to generate a new learning model using deep learning or to generate a new learning model by transfer learning using an existing general-purpose learning model. The deep learning unit 204 performs such a determination, for example, when the detection assistance device 100A selects a learning model.

For example, consider a case where the deep learning unit 204 generates a gaze feature learning model in a situation where a large amount of learning content images can be prepared. In this case, gaze information of an expert is acquired for each learning content image, and a set of correct answer contents (learning data) is generated. After that, the deep learning unit 204 trains the deep learning model (estimation model) by machine learning using the set of learning content images and the set of correct answer contents, that is, generates a gaze feature learning model by new learning.
On the other hand, when generating a gaze feature learning model in a situation where a large amount of learning content images cannot be prepared, the deep learning unit 204 generates the gaze feature learning model by transfer learning of a gaze feature learning model corresponding to another learning content image that has already been generated by deep learning. Note that whether or not a large amount of learning content images can be prepared is determined, for example, according to the number of learning content images stored in the learning content image DB 205 or according to a selection operation by the user.
When generating a gaze feature learning model by new learning, the deep learning unit 204 executes the process shown in step S65. When generating a gaze feature learning model by transfer learning, the learning device 200 executes the process shown in step S62.

Step S62:
The deep learning unit 204 selects a predetermined learning model from among the learning models stored in the gaze feature learning model DB 208. For example, for a set of learning content images selected by a user, the deep learning unit 204 selects a learning model that has been trained for other sets of learning content images. The deep learning unit 204 acquires the selected learning model as a deep learning model to be used for transfer learning.
Step S63:
The deep learning unit 204 extracts, from the deep learning model read out in step S62 for use in transfer learning, the layers from the input layer to the intermediate layer (adaptation layer) designated by the user or designated in advance as a transfer learning model. The deep learning unit 204 then extracts the intermediate layers subsequent to the adaptation layer from the deep learning model, connects them to the adaptation layer of the transfer learning model, and connects the output layer to configure a deep learning model for transfer learning.
Step S64:
The deep learning unit 204 performs optimization processing of the weight parameters of the layers of each network so that, when a pixel determined to have a high degree of attention based on gaze information of an expert in a learning content image is input to the input layer of the learning target model (the deep learning model for transfer learning or the deep learning model), a numerical value close to "1" indicating a high degree of attention is output from the output layer. Also, the deep learning unit 204 performs optimization processing so that, when a pixel determined to have a low degree of attention of an expert in a learning content image is input to the input layer of the learning target model, a numerical value close to "0" indicating a low degree of attention is output from the output layer. That is, the deep learning unit 204 performs machine learning of class classification on the learning content image and generates a gaze feature image as a learning result.

At this time, the deep learning unit 204 may input a set of learning content images different from the learning content and a set of correct answer data which is gaze information of an expert on those images to the generated learning model, and conduct a learning test on the generated learning model.
In this case, when the deep learning unit 204 inputs a collection of learning content images into the learning model, if the numerical value output from the output layer is greater than or equal to a predetermined first threshold and the numerical value output from the output layer is less than or equal to a predetermined second threshold, the deep learning unit 204 stores this learning model in the gaze feature learning model DB 208 and sets it as the gaze feature learning model.
On the other hand, if, in the above-mentioned learning test, the value output by the output layer of the learning model for a pixel on which the expert's gaze is focused is less than a preset first threshold, or the value output by the output layer of the model to be learned for a pixel on which the inspector's gaze is not easily focused is equal to or greater than a preset second threshold, the deep learning unit 204 does not store this learning model in the gaze feature learning model DB 208 and re-learns the learning model.

Step S65:
From the learning model generated in step S64, the deep learning unit 204 extracts, as parameters of the learning model, the output parameters of the pooling layer and convolution layer in the intermediate layer of the multi-layer structure, the type of activation function and the output parameters, etc.

Step S66:
The deep learning unit 204 stores the generated learning model and the extracted learning model parameters in the gaze feature learning model DB 208 (registration process).

Third Embodiment
Next, a third embodiment will be described. This embodiment differs from the above-mentioned embodiment in that a content feature amount is calculated using visual features and gaze features. As a result, the detection support device 100B of this embodiment can perform processing similar to information processing of human visual perception and use human gaze information, and can present more detailed information about how the content image appears. In this embodiment, only the configuration different from the above-mentioned embodiment will be described, and the same reference numerals will be used for the configuration similar to the configuration of the above-mentioned embodiment, and the description thereof will be omitted unless particularly necessary.

FIG. 11 is a block diagram showing an example of the configuration of a detection support device 100B according to the third embodiment. The detection support device 100B includes, for example, a visual feature/gaze feature calculation unit 119 and a content feature amount calculation unit 105B.

In the following, in this embodiment, the content image is described as a still image, but similar to the first embodiment, this may be applied to other content images such as moving images and video.

The visual feature/gaze feature calculation unit 119 calculates the visual feature/gaze feature using the visual feature image and the gaze feature image. The visual feature/gaze feature is information indicating the degree of both the visual feature and the gaze feature. The visual feature/gaze feature calculation unit 119 calculates the visual feature/gaze feature by, for example, performing an operation between the visual feature and the gaze feature. The operation here includes, for example, various logical operations such as AND, OR, and exclusive OR (XOR) between the visual feature and the gaze feature, as well as winners take all operations, bit operations, and arithmetic operations.

When performing calculations between visual features and gaze features, the visual feature/gaze feature calculation unit 119 may weight each feature or each pixel.

The visual feature image used by the visual feature image calculation unit 119 is an image generated by the visual feature image generation unit 103. The visual feature image used by the visual feature image calculation unit 119 is an image generated by the gaze feature image generation unit 117. The visual feature gaze feature calculation unit 119 outputs the generated visual feature gaze feature to the content feature amount calculation unit 105A.

The content feature amount calculation unit 105B calculates the content feature amount by applying the image feature to the visual feature/gaze feature. The content feature amount in this embodiment is, for example, a statistical amount that statistically indicates the image features in the visual feature/gaze feature.

For example, when contrast is selected as the image feature, the content feature amount is a value indicating what kind of contrast the visual feature/gaze feature forms in the content image. The visual feature/gaze feature used here is information generated by the visual feature/gaze feature calculation unit 119. The image feature used here is the image feature selected by the image feature selection unit 104.
The content feature amount may be the visual feature or the gaze feature itself.

Figure 12 is a flowchart showing an example of the operation of the processing performed by the detection support device 100B according to this embodiment. The processing shown in each of steps S31, S32, S36, and S38 to S41 in Figure 12 is similar to the processing shown in each of steps S11, S12, S13, and S15 to S18 in Figure 2, and therefore a description thereof will be omitted. In addition, the processing shown in steps S33 and S34 in Figure 12 is similar to the processing shown in steps S21 and S22 in Figure 2, and therefore a description thereof will be omitted.

Step S30:
The detection support device 100B outputs the content image acquired by the content image selection unit 101 to the visual feature image generation unit 103 and the gaze feature image generation unit 117.
Step S35:
The detection support device 100B calculates visual feature gaze features using the visual feature image generated in S32 and the visual feature image generated in step S34, and outputs the calculated visual feature gaze features to the content feature calculation unit 105B.
Step S37:
Detection support device 100B calculates content feature amounts by applying the image features to the visual feature and gaze feature calculated in step S35 using content feature amount calculation unit 105B.

As described above, the detection support device 100B of the third embodiment includes a visual feature gaze feature calculation unit 119. The visual feature gaze feature calculation unit 119 calculates visual feature gaze features using a visual feature image and a gaze feature image. As a result, the detection support device 100B of the third embodiment can perform processing similar to the information processing of human visual perception and utilize human gaze information, making it possible to present more detailed information about how the content image appears.

In addition, a program for realizing all or part of the functions of the detection assistance device 100 (100A, 100B) of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.
Additionally, "computer system" includes a WWW system equipped with a homepage providing environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that hold a program for a certain period of time, such as volatile memory (RAM) within a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and gist of the invention.

Reference Signs List 100, 100A, 100B... Detection support device 101... Content image selection unit 102... Visual feature selection unit 103... Visual feature image generation unit 104... Image feature selection unit 105, 105A, 105B... Content feature amount calculation unit 106... Analysis method selection unit 107... Analysis unit 108... Content image DB
109...Visual feature DB
110: Visual feature image storage unit 111: Image feature DB
112: Content feature amount storage unit 113: Analysis method DB
114: Analysis result storage unit 115: Analysis result output unit 116: Gaze feature learning model selection unit 117: Gaze feature image generation unit 118: Gaze feature learning model DB
119...visual feature and gaze feature calculation unit

Claims (10)

A detection support device that supports detection of the presence or absence of an unexpected pattern due to continuity of a pattern in a content image generated by repeatedly arranging a predetermined pattern, the detection support device comprising:
a content image acquisition unit for acquiring the content image;
a gaze feature image generating unit that estimates gaze features that are characteristics of a gaze when detecting the pattern from the content image by using a gaze feature learning model, and generates a gaze feature image that indicates the estimated gaze features for each pixel in the content image;
an analysis result output unit that displays the gaze feature image;
Equipped with
The gaze feature learning model is a trained model that has learned a correspondence between the content image for learning and the gaze feature of a person who detected the pattern from the content image for learning.
Detection aids. a visual feature image generating unit for generating a visual feature image representing a visual feature, which is a feature of brightness or color, from the content image;
a visual feature gaze feature calculation unit that calculates the visual feature and the gaze feature using the visual feature image and the gaze feature image, and calculates a result of the calculation as a visual feature gaze feature;
Further equipped with
The analysis result output unit outputs the visual feature/gaze feature as a gaze and visual feature when detecting the pattern in the content image.
The detection assistance device according to claim 1 . the visual feature image generating unit generates the visual feature image by further using a recognition index representing an ease of recognition of the content image by a human eye as the visual feature.
The detection assistance device according to claim 2 . The recognition indices include at least one of a visual attention model, a gaze prediction model, a saliency model, and a saliency model.
The detection assistance device according to claim 3 . A learning device that generates a gaze feature learning model to be used by a detection support device that supports detection of the presence or absence of an unexpected pattern due to the continuity of a pattern in a content image generated by repeatedly arranging a predetermined pattern, the device comprising:
the gaze feature learning model is a trained model that has learned a correspondence relationship between the content image for learning and a gaze feature that is a feature of the gaze of a person who detected the pattern from the content image for learning,
a deep learning unit configured to be able to select either deep learning or transfer learning using an existing learning model as a method for generating the gaze feature learning model, and to generate the gaze feature learning model using the selected method;
A learning device comprising: a learning content image database for storing the learning content image;
The deep learning unit determines whether to select deep learning or transfer learning using an existing learning model depending on the number of the content images for learning stored in the learning content image database.
The learning device according to claim 5 . A detection support method performed by a computer, which is a detection support device, for supporting detection of the presence or absence of an unexpected pattern due to the continuity of a pattern in a content image generated by repeatedly arranging a predetermined pattern, the method comprising:
a content image acquisition unit acquires the content image;
a gaze feature image generating unit that estimates gaze features that are characteristics of a gaze when detecting the pattern from the content image using a gaze feature learning model, and generates a gaze feature image that indicates the estimated gaze features for each pixel in the content image;
an analysis result output unit displays the gaze feature image;
The gaze feature learning model is a trained model that has learned a correspondence between the content image for learning and the gaze feature of a person who detected the pattern from the content image for learning.
Detection aid methods. A program for causing a computer to function as the detection support device described in claim 1. A learning method performed by a computer, which is a learning device, for generating a gaze feature learning model used by a detection support device that supports detection of the presence or absence of an unexpected pattern due to the continuity of a pattern in a content image generated by repeatedly arranging a predetermined pattern, the method comprising:
the gaze feature learning model is a trained model that has learned a correspondence relationship between the content image for learning and a gaze feature that is a feature of the gaze of a person who detected the pattern from the content image for learning,
The deep learning unit is configured to be able to select either deep learning or transfer learning using an existing learning model as a method for generating the gaze feature learning model, and generates the gaze feature learning model using the selected method.
How to learn. A program for causing a computer to function as the learning device described in claim 5.

Images