JP2023177010A - Image analysis method, image analysis device and image analysis program - Google Patents

Abstract

To reduce a load of the work of a specialist in a data survey in forensic, shorten the necessary time and improve efficiency.SOLUTION: An analysis method comprises steps of: extracting image data from a data group including a plurality of pieces of text data and a plurality of pieces of image data; acquiring the extracted image data; and classifying a class of the acquired image data by using a learned model that has learned the plurality of pieces of image data and teacher data associated with any of a first character class including a handwritten character, a second character class including a character other than hand writing, and an image class in which each image data does not include the character as the class of each image data.SELECTED DRAWING: Figure 3

Description

Application for application of Article 30, Paragraph 2 of the Patent Act (1) Website posted on November 27, 2021 https://www. pwc. com/jp/ja/services/forensic/forensic-image-analyzer. html

The present invention relates to an image analysis method, an image analysis device, and an image analysis program for analyzing image data included in an image data group.

Currently, in data investigation in digital forensics, character data is targeted for digital analysis such as word frequency analysis because the amount of data handled is enormous. On the other hand, image data is currently difficult to analyze, such as word frequency analysis. However, image data can also contain information useful for data investigation. Therefore, when using image data for data investigation, data investigation experts must visually check all of the huge amount of image data, process all image data with OCR (Optical Character Recognition), It was necessary to perform digital analysis on the obtained character data. Visually checking all image data places a heavy workload on experts and requires a great deal of time. In addition, when performing OCR processing on all image data, image data that has not been properly OCR processed is also OCR processed once, so in addition to the hassle of OCR processing, the data that was not OCR processed must be re-processed by an expert. Currently, it is not possible to sufficiently reduce the burden on experts because visual confirmation is also required.

In recent years, technology for converting data into images has become widespread. For example, cameras installed in mobile terminals are becoming more sophisticated, and the images that can be taken with these cameras are also becoming higher resolution. Therefore, although the amount of image data has been increasing in recent years, it is expected that the number of image data containing important information will further increase in the future. Therefore, it is desirable to be able to easily use image data for forensic data investigation.

For example, Patent Document 1 describes a document analysis system that extracts and presents types of necessary information. This document analysis system analyzes digital information for use in litigation or fraud investigations. As described above, conventional document analysis systems sometimes analyze documents using keywords, but do not target images.

Patent No. 5596213

In view of the above, the present invention provides an image analysis method, an image analysis device, and an image analysis program that easily utilize image data for forensic data investigation.

An image analysis method according to the present disclosure includes a step of extracting image data from a data group including a plurality of character data and a plurality of image data, a step of acquiring the extracted image data, a plurality of image data, As the class of each image data, teacher data is associated with one of the following: a first character class containing handwritten characters, a second character class containing characters other than handwritten characters, or an image class in which each image data does not contain characters. The method includes the step of classifying the acquired image data into classes using the trained model.

According to the image analysis method, image analysis device, and image analysis program of the present disclosure, image data can be easily used for data investigation in forensics.

FIG. 2 is a schematic diagram showing a forensic procedure. FIG. 2 is a schematic diagram showing a data group targeted by the image analysis method of the present disclosure. FIG. 1 is a block diagram showing the configuration of an image analysis device of the present disclosure. FIG. 2 is a schematic diagram illustrating a learned model used in the image analysis device of the present disclosure. FIG. 2 is a schematic diagram illustrating learning of a learned model used in the image analysis device of the present disclosure. FIG. 4B is a schematic diagram showing an example of classes that can be classified using the learned model of FIG. 4A. FIG. 4B is a schematic diagram showing another example of classes that can be classified using the trained model of FIG. 4A. 3 is a flowchart showing processing of the image analysis device of the present disclosure. It is a block diagram showing the composition of the image analysis device concerning a modification. It is a flow chart which shows processing of an image analysis device concerning a modification.

An image analysis method, an image analysis device, and an image analysis program according to embodiments will be described below with reference to the drawings. In the present disclosure, an image analysis method, an image analysis device, and an image analysis program analyze image data used for data investigation in forensics. In the following description, the same configurations will be designated by the same reference numerals and the description will be omitted.

<Overview of digital forensics>
In recent years, digital forensics has been faced with the issue of an increase in the amount of data that is subject to forensic investigation. In e-discovery (electronic evidence discovery), which is an example of where digital forensics is used, as shown in the schematic diagram of Figure 1, "evidence preservation" is used to prevent evidence from being tampered with, and "evidence preservation" is used to preserve evidence while preserving evidence. Documents related to electronic evidence are created through a series of processes such as "collection" of data, "processing" of data for data review and analysis as necessary, "review" of data, and "analysis" of data. Generate and submit. At this time, the challenge is to implement the process of extracting data to be presented as evidence as needed quickly and with high precision, and to improve efficiency. In addition, for example, when a third-party investigation committee is established in a corporate fraud case and digital forensics is used in the investigation, due to costs and other factors, in order to report to stakeholders such as shareholders, Sometimes it is necessary to produce research results within a very short research period. Therefore, the problem of increasing amounts of data that is subject to forensic investigations has an extremely large impact. Currently, it is said that about 70% of the cost of e-discovery is spent on the document "review" process, and the challenge is to simplify and improve the efficiency of the review process.

In digital forensic investigations, vast amounts of digital data stored in recording media built into multiple information processing terminals such as personal computers and smartphones, as well as multiple external storage media, etc., are investigated. It is common for the amount of data in the target data group to exceed several terabytes in a single project, and it is extremely difficult to conduct a visual inspection alone when carrying out the series of processes described above using Figure 1. be. Therefore, for example, for "text data" such as document files and text files that can be analyzed by word character and keyword search, digital methods such as classifying them using keyword searches or extracting data with high importance can be used. can be analyzed. On the other hand, since it is difficult to easily perform character analysis or keyword searches on "image data," visual inspection is the main method. In recent years, as data scanning, screen capture, and image capture using digital cameras and smartphones have become commonplace, the amount of image data is increasing, and the amount of image data that is subject to forensic investigations is also increasing. are doing. For example, the number of image data is often 100,000 or more, and human visual inspection takes an enormous amount of time, making it impractical for limited period investigations. In the image analysis device and image analysis method according to the present disclosure, by classifying image data to be subjected to forensic investigation in advance, it becomes possible to easily use the image data for data investigation. As a result, by using the image analysis device and the image analysis method according to the present disclosure, a series of digital forensic processes can be performed quickly and with high accuracy, and can be made more efficient. Note that the image analysis device and image analysis method according to the present disclosure can streamline the "review" process using image data by classifying the image data in the "processing" process shown in FIG. .

<Explanation of terms>
In the present disclosure, "data" refers to digital data, which is data expressed in binary numbers that can be processed by a computer. Furthermore, in the following description, as shown in FIG. 2, data will be explained as being able to be classified into "text data,""imagedata," and "data other than text and images."

"Character data" is digital data that indicates characters using a numerical value string assigned in advance. Examples of character data are document files such as word files, text files, and the like. For example, character data is data that can be subjected to character analysis or keyword search.

"Image data" is digital data that indicates information by assigning numerical values to a plurality of pixels. Although its format is not limited, it may be a binary image, a grayscale image, or a color image (RGB image). For example, the image data may be normal photo data taken with a camera, data obtained by screen capturing the display of a personal computer or smartphone, data obtained by converting a printed document into an image, or the like. Typical image data formats include jpeg data, png data, tif data, and gif data.

"Image data containing characters" mainly means that the data information includes main characters. For example, photo data of a whiteboard with characters written on it can be image data that includes characters. Furthermore, data obtained by screen capturing the display of a computer or smartphone on which chat and SNS history is displayed can be image data. Furthermore, data obtained by converting document data into PDF can also be image data. Image data including these characters is data in which the characters included in the image data can be an important element of the data. Therefore, for example, if the image data is a landscape photograph and includes a signboard with text written on it, the image data may not correspond to image data that includes text.

"Image data including characters other than handwritten characters" is, for example, image data including characters expressed in a specified font. This "regular font" refers to a typeface with high accuracy of character recognition. For example, photographic data of a printed book or the like can be image data that includes characters other than handwritten characters because the characters contained therein are in a specified font. Further, document data created on a personal computer that is converted into PDF may be image data that includes characters other than handwritten characters. Furthermore, screen-captured data of a computer or smartphone display that displays chat and SNS history can be image data that includes characters other than handwritten characters.

"Image data including handwritten characters" is, for example, image data including characters written by a human. For example, photo data of a whiteboard or notebook on which characters are written by a human can be image data that includes handwritten characters. Further, data in which characters written by a human being are converted into PDF may be image data including handwritten characters. Note that even if the characters are not written by a human, an image that includes characters printed in a handwritten style can be determined to be image data that includes handwritten characters.

Here, if the image data includes both "handwritten characters" and "non-handwritten characters", compare the ratio of the number of characters between handwritten characters and non-handwritten characters, and find that the ratio of handwritten characters is higher. In this case, the image data may include handwritten characters, and if the proportion of non-handwritten characters is large, the image data may include non-handwritten characters.

"Image data that does not include text" is, for example, image data that mainly contains image information such as landscape photographs and portrait photographs.

"Data other than text or images" is, for example, data such as audio files, CAD files, spreadsheet files, presentation files, and spreadsheets.

The "data group" is a set of data including "character data", "image data", and "data other than characters or images" as described above with reference to FIG.

"Forensics" refers to the collection and analysis of data records, etc. in criminal investigations and legal disputes.

“Forensic data” refers to data that can be the subject of investigation in, for example, criminal investigations or legal disputes. For example, forensic data may include data that has been determined to be evidence, even if it is not actually used as evidence in discovery in a lawsuit. In other words, forensic data is data that constitutes a data group for searching for evidence data.

<Image analysis device>
The image analysis device according to the embodiment is a device that extracts and analyzes data that meets conditions from forensic data, which is a data group used in a specific investigation. The data group considered to be forensic data is a huge amount of data, and it is difficult to investigate it easily even if a large number of personnel are introduced. Therefore, the image analysis device according to the embodiment classifies each image data included in the data group into a predetermined class using image analysis. Furthermore, in order to facilitate investigation, the image analysis device may assign a class to image data and present it to an expert or the like as a classification result. The class of this image data can indicate data attributes such as whether the image includes characters, for example. Furthermore, the image analysis device can use the classification results to select data useful for the investigation from the data used in a specific investigation and present it to an investigation expert or the like. Here, the data selected as data useful for the investigation is specific data that has a high possibility of being used as evidence in a digital forensic investigation, in other words, data that has a high priority as evidence. A specific example of data with high priority will be described later.

The image analysis device 1 according to the embodiment will be described using FIG. 3. For example, the image analysis device 1 is realized by an information processing device including a control section 10, a storage section 20, a communication section 30, an input section 40, and an output section 50, as shown in FIG.

The communication unit 30 is a communication means for enabling data communication with an external device (for example, a storage medium that stores a data group). Data communication may be performed wirelessly and/or by wire according to known communication standards. For example, wired data communication is performed by using, as the communication device 22, a communication controller of a semiconductor integrated circuit that operates in accordance with the Ethernet (registered trademark) standard and/or the USB (registered trademark) standard. In addition, wireless data communication operates in accordance with the IEEE802.11 standard for LAN (Local Area Network) and/or the 4th/5th generation mobile communication system, so-called 4G/5G, for mobile communication. This is done by using a communication controller of a semiconductor integrated circuit as the communication device 22.

The input unit 40 is an input means such as an operation button, a keyboard, a mouse, a touch panel, a microphone, etc. used for requesting image analysis and inputting data. Further, the output unit 50 is an output means such as a display, a speaker, etc. used for outputting processing results and data.

The storage unit 20 is a recording medium that records various information. The storage unit 20 is realized by, for example, a RAM, a ROM, a flash memory, an SSD (Solid State Drive), a hard disk drive, another storage device, or an appropriate combination thereof. The storage unit 20 stores an image analysis program P, which is a computer program executed by the control unit 10, as well as data used in image analysis processing, various data obtained in image analysis processing, and the like.

The control unit 10 is a controller that controls the entire image analysis device 1 . The control unit 10 executes the image analysis program P stored in the storage unit 20 to perform processing as the extraction unit 101, classification unit 102, conversion unit 103, analysis unit 104, and output processing unit 105. Execute. The control unit 10 is not limited to a configuration in which a predetermined function is realized by cooperation of hardware and software, but may be a hardware circuit designed exclusively for realizing a predetermined function. That is, the control unit 10 can be realized by various processors such as a CPU, MPU, GPU, FPGA, DSP, and ASIC.

The image analysis program P includes a learned model M. Using this trained model M, the image analysis device 1 can classify and output the class of image data that is input data, as shown in FIG. 4A. This trained model M has at least two classes of image data: a class of data with a high priority as evidence in forensics, and a class of data with a lower priority as evidence than the class of data with a high priority. When two classes are set, as shown in FIG. 4B, the learning device has already learned the relationship between a plurality of image data and the class associated with this image data. Note that the number of classes that can be classified is not limited and can be determined depending on the forensic target.

The storage unit 20 stores, in addition to the image analysis program P, data used in image analysis and various data obtained in image analysis processing. For example, the storage unit 20 stores image data 202, class data 203, text data 204, and analysis result data 205.

The image data 202 is data extracted from a data group as a target of image analysis in the image analysis device 1.

Class data 203 is data indicating the class of each image data as a classification result of the image data 202.

Text data 204 is data obtained by converting character information included in image data 202.

Analysis result data 205 is data indicating the result of character analysis of text data 204. For example, the analysis result data 205 includes the quantity of specific words included in each text data 204 in association with the identification information of each text data 204.

The extraction unit 101 extracts image data from a data group including a plurality of character data and a plurality of image data. This data group is forensic data used in a specific investigation. Specifically, the data group is data stored in a storage device (not shown) in the preservation and collection of evidence described above with reference to FIG. 1 for forensic purposes, for example. The storage device that stores the data group may be the storage unit 20 of the image analysis device 1, or may be an external device with which the image analysis device 1 can communicate data via a network (not shown). .

Further, the extraction unit 101 stores the extracted image data 202 in the storage unit 20 as an analysis target. At this time, the extraction unit 101 may store list data of identification information of the extracted image data 202 in the storage unit 20 as image data, instead of storing the extracted image data itself in the storage unit 20. In this case, the image analysis device 1 accesses the image data stored in the external storage device via the network based on the identification information stored in the storage unit 20 at the timing when the image data 202 is required in later processing. However, the configuration may be used. For example, when e-mail data includes image data as an attached image along with text data in the main text, the extraction unit 101 can treat the text data and image data as separate data, and You may also extract the image data that is displayed. Furthermore, when the data of the document file includes image data as a pasted image along with the character data of the main text, the extraction unit 101 makes it possible to treat the character data and image data as separate data. You may also extract the image data that is displayed. Regarding data that is a combination of text data and image data as described above, the extraction unit 101 can handle the text data and image data separately, and may extract only the image data.

The classification unit 102 uses the trained model M to classify the classes of image data. For example, as shown in FIG. 5A, the learned model M includes a plurality of image data and classes of each image data, such as a "first character class" that includes handwritten characters, and a "second character class" that includes characters other than handwritten characters. This is a model that has already learned training data that is associated with either "character class" or "image class" that does not contain characters. Therefore, by using the trained model M that has trained the teacher data including image data of each class shown in FIG. 5A, if the input image data is a photo of a whiteboard containing handwritten characters, the corresponding Image data is classified into "first character class". Furthermore, when image data such as a screen capture image of a chat or a scanned image of a printed document that includes characters other than handwritten characters is input, the trained model M classifies the data into the "second character class." Furthermore, when image data that is a landscape photograph that does not include text is input, the trained model M classifies it into an "image class." Further, the classification unit 102 generates class data 203 in which the identification information of the image data is associated with the identification information of the classified class as an attribute as a classification result, and stores it in the storage unit 20.

If the image data that is the teacher data includes at least both "handwritten characters" and "non-handwritten characters," the class label of the image data may be set as "first character class." For example, when handwritten characters and non-handwritten characters are included, it can be assumed that the image data is saved because the handwritten characters are important. Therefore, it is considered important to treat handwritten characters as the main image data. By using the trained model M generated using such teacher data, when input image data includes at least both "handwritten characters" and "non-handwritten characters", the image data can be Classified as "first character class".

Alternatively, suppose that the image data that is the training data includes at least both "handwritten characters" and "non-handwritten characters", and the ratio of the number of handwritten characters included in the image data is greater than the number of non-handwritten characters. When the ratio is higher than the percentage (for example, when it is larger than 50%), the class of the image data may be set as the "first character class." Conversely, when the image data that is the teacher data contains handwritten characters in a lower proportion than the proportion of non-handwritten characters (for example, less than 50%), the image data The class may be set as the "second character class". For example, when the image data includes handwritten characters and non-handwritten characters, it can be assumed that the image data is saved because information with a large number of characters is important. Therefore, it is considered important to handle handwritten or non-handwritten characters with a large number of characters as the main image data. By using the trained model M generated using such training data, the input image data includes at least both "handwritten characters" and "non-handwritten characters" and the number of handwritten characters is reduced. , than the number of non-handwritten characters, the image data is classified as "first character class". By using the trained model M generated using such teacher data, when the proportion of "handwritten characters" included in the input image data is higher than the proportion of "non-handwritten characters", the corresponding The image data is classified as "first character class." On the other hand, when the percentage of "handwritten characters" included in the input image data is lower than the percentage of "non-handwritten characters," the image data is classified as a "second character class."

In addition, the "second character class" is a class of data that has a high priority as evidence in forensics, and the "third character class" has a lower priority as evidence than the data of the third character class. A "fourth character class" which is a data class can be included. For example, as shown in FIG. 5B, the third character class includes screen capture images of chats, screen capture images of instant messages, screen capture images of SNS posts, screen capture images of short messages (SMS) from mobile phones, etc. is attached to the image data. Further, the fourth character class is assigned to image data such as a scanned image of a printed document or a captured image of a web page.

For example, "printed documents" and "web pages," which fall under the fourth character class, are information that may need to be referenced repeatedly. It is also common to be saved as . On the other hand, "chat", "instant message", "SNS posting", etc. that fall under the third character class are generally used for temporary exchange of information and are usually not read. They are rarely preserved separately after the Therefore, the fact that "chat", "instant message", "SNS post", etc., which are not normally saved, is recorded as image data indicates that the information is important and worth preserving. It can be assumed that it was considered. Therefore, it is assumed that the image data corresponding to the third character class has a higher priority as evidence than the image data corresponding to the fourth character class. Therefore, in image data that includes characters other than handwritten characters, the class of data that has a high priority as evidence in forensics is defined as the "third character class", and the class of data that has a lower priority as evidence than the third character class as " 4th character class".

In this way, when the second character class includes the third character class and the fourth character class, the training data set of the trained model M includes the "first character class", "third character class" It includes image data that is teacher data associated as a label of "Class", "Fourth Character Class", or "Image Class". In addition, when using a trained model M that has been trained using image data of "first character class," "third character class," "fourth character class," and "image class" as training data, the classification unit 102 is classified into one of these four classes.

The conversion unit 103 refers to the image data 202 and the class data 203 in the storage unit 20, performs character recognition processing only on the image data associated with a specific class, converts each into text data 204, and stores the image data in the storage unit 20. 20 to be memorized. Specifically, the conversion unit 103 converts character information included in the image data 202 into text data 204 by OCR processing.

At this time, the conversion unit 103 executes character recognition processing only on the image data 202 associated with the second character class, and converts character information included in the image data 202 associated with the second character class. It may be converted as text data 204 and stored in the storage unit 20. For example, if the image data 202 including handwritten characters classified into the "first class" is preferably visually checked by an expert, such as when sufficient character recognition accuracy cannot be obtained, the "second class" Character recognition processing is performed only on image data 202 classified into "class". Further, for example, the content of character information included in the image data 202 classified into the "fourth character class" can be easily understood through analysis by the analysis unit 104, which will be described later; If the classified image data 202 is undesirable as an analysis target in the analysis unit 104 (described later) due to the frequent use of abbreviations, only the image data 202 classified into the "fourth character class" is subjected to character recognition. It may also be a processing target.

The analysis unit 104 analyzes the characters included in the image data by determining the number of specific words included in the text data 204. For example, the analysis unit 104 can analyze words included in the text data 204 using morphological analysis and word frequency analysis. The analysis unit 104 may also cause the storage unit 20 to store the quantity of specific words included in the text data 204 obtained by the conversion in the conversion unit 103 as analysis result data 205 in association with the identification information of the image data. good. For example, the "quantity of specific words" is either the number of specific words included in the data or the proportion of the specific words in the number of words in the entire data. This makes it possible to infer the content of character strings included in image data and use it for review and document generation decisions.

Note that the number of specific words is not limited, and a plurality of words can be targeted. Furthermore, when a plurality of numbers of specific words are determined, the quantity may be determined for each word. Furthermore, a group may be provided for each word with the same or similar meaning, and the number of words may be determined for each of a plurality of words. A plurality of words included in this group may be determined for each target investigation. By targeting a plurality of words and finding the quantity for each word or for each group of words, the content of the character string included in the image data 202 can be easily estimated.

The output processing unit 105 outputs the classification result by the classification unit 102, the conversion result by the conversion unit 103, and/or the analysis result by the analysis unit 104 to the output unit 50. At this time, the output processing unit 105 can output the analysis result, conversion result, and/or analysis result in association with the image data 202. Note that the method of output is not limited, and in addition to outputting to the output unit 50 such as a display or printer, data may also be output to an external information processing device that can communicate data via the communication unit 30. good.

- Output of analysis results (ranking) For example, the output processing unit 105 can output the image data 202 in association with a ranking according to the number of specific words included in the text data 204. Alternatively, when a group including a plurality of words is set, the output processing unit 105 may output the image data 202 in association with the ranking determined for this group. By ranking and outputting specific words according to the number of specific words included in the image data 202, it becomes easier for experts to guess the contents of character strings included in the image data.

Note that in this case as well, the number of specific words is not limited, and a plurality of words can be targeted. Further, when a plurality of numbers of specific words are determined, different weights may be set for each word among the plurality of words. That is, words that are likely to have an influence on guessing the content of a character string may be given a particularly high weight. By setting different weights for each word and calculating the number of words based on the total to determine the ranking, it is possible to more easily guess the content of the character string included in the image data 202.

- Output of image data and classes (attributes) For example, when outputting the image data 202 and/or when outputting the identification information of the image data 202, the output processing unit 105 outputs the associated information for each image data 202. Attributes indicating the class can also be output. Specifically, in the class data 203, the output processing unit 105 selects "first character class", "third character class", "fourth character class", as classes indicated for each image data 202, Output by associating classes such as "image class". Thereby, the expert conducting the investigation can confirm the image data 202 and at the same time instantly determine what processing will be necessary for the image data 202 in the future.

For example, as shown in FIG. 5A, in an example where "first character class", "second character class", and "image class" are set, as described above, the image of "second character class" It is assumed that the data 202 is converted into text data 204 by the conversion unit 103 and analyzed by the analysis unit 104. At this time, the expert uses the analysis result data 205 without visually inspecting the image data 202 of the "second character class", and uses the analysis result data 205 to Rules such as making only the data 202 subject to visual inspection can be determined. This makes it possible to make the work of experts more efficient.

For example, in an example classified into "first character class", "third character class", "fourth character class", and "image class", as described above, "fourth character class" In a situation where the image data 202 of the "third character class" is judged to have higher priority as evidence than the image data 202 of It may also be the subject of It is also possible to output only the image data 202 of the "third character class". Alternatively, only the "third character class" and "fourth character class" image data 202 may be output. This makes it possible to prioritize and analyze important image data that is likely to be used as evidence in a trial out of a huge amount of image data of over 100,000 items, making the work of experts more efficient. . It would take a huge amount of time for humans to visually judge and analyze all 100,000 pieces of image data, so it would be impractical for a fraud investigation by a third-party committee, which has a limited investigation period. Therefore, a technique necessary for forensic investigations is to extract only important image data without leaking it. In particular, "chat", "instant message", "SNS posting", etc. that fall under the third character class can be used as evidence of instructions directly related to the determination of fraud in a corporate fraud case, etc. It often happens. Therefore, by outputting only the image data 202 of the third character class, or only the image data 202 of the third character class and the image data 202 of the fourth character class from a huge amount of image data, it is possible to At this stage, a conclusion can be drawn as to whether or not important evidence is found. This is especially effective in fraud investigations conducted by third-party committees, which typically have to analyze large amounts of data and prepare investigation reports under time constraints. In particular, when outputting only the image data 202 of the "third character class" (image data of "chat", "instant message", "SNS post", etc.) which is very rarely saved, By extracting only a very limited number of images from the entire image data, image data that can serve as important evidence in a trial can be obtained at the very early stage of the investigation period.

- Select and output by class In addition, the output processing unit 105 can select a class associated with image data as a filter and output it to the output unit 50. Specifically, when display of image data 202 of a specific class is requested via the input unit 40, the output processing unit 105 selects the image data 202 of the requested class and outputs it to the output unit 50. can be done. Further, when display of image data 202 including a specific word is requested via the input unit 40, the output processing unit 105 selects the image data 202 including this specific word using this specific word as a filter. It can be outputted to the output section 50. Here, when the request includes a quantity of words in association with a specific word, the output processing unit 105 can select the image data 202 that includes the specific word in a specific quantity and cause the output unit 50 to output the selected image data 202 . This makes it easy for the expert to visually inspect only the necessary image data 202, thereby making the work more efficient.

- Output text data along with image data and analysis results (quantity, ranking) The output processing unit 105 outputs the quantity of specific words included in the calculated text data 204 and the text data 204 stored in the storage unit 20 The text data 204 and the image data 202 may be output in association with a ranking according to the number of specific words included in the text. This makes it easier for experts to guess the content of the image data 202, so they can work more efficiently.

- Output as a list The output processing unit 105 may generate and output list data that is a list of the image data 202 stored in the storage unit 20. Specifically, the output processing unit 105 associates the file name of the image data 202 with an attribute, which is a class into which the image data 202 is classified, and outputs the associated file name. This makes it easier for the expert to estimate the overall structure of the target image data 202 included in the data group, so that the expert can work more efficiently. Alternatively, only the image data 202 of the "third character class" and the "fourth character class" may be output. Furthermore, regarding the order of the output list, the image data 202 of the "third character class" may be outputted in the highest order relative to other character classes. In addition, regarding the order of the output list, "1. Image data of the third character class", "2. Image data of the fourth character class", "3. Image data of the first character class", "4. The data may be output in the order of "image data of image class." This makes it possible to prioritize and analyze important image data that is likely to be used as evidence in court from among a huge amount of image data of more than 100,000 items, streamlining the work of experts. . In particular, by outputting image data of the third character class from a huge amount of image data in the order of the top of the output list, it is possible to reach a conclusion as to whether or not important evidence can be found at the early stage of a forensic investigation. can. This is highly effective in fraud investigations conducted by third-party committees, which typically have to analyze large amounts of data and prepare investigation reports under time constraints. Furthermore, when "image data" and "character data" are output as an output list, "image data of the third character class" can also be output as a higher rank than "character data." Normally, "character data" is often output with priority over image data. However, by prioritizing "image data of the third character class" over "text data" and outputting it in a higher order than "text data," it can be used as evidence in a trial at the initial stage of a forensic investigation. Able to draw conclusions about the presence or absence of likely material evidence.

Note that in the example described above using FIG. 3, the image analysis device 1 is represented by one computer, but is not limited to this. The image analysis device 1 may be composed of multiple computers. For example, any of the processing in the extraction unit 101, the classification unit 102, the conversion unit 103, or the analysis unit 104 may be executed by an external device connected via a network. Further, for example, some of the data described above as being stored in the storage unit 20 can be stored in another device connected via a network, and the image analysis device 1 can read data from an external device, It may also be configured to store data in an external device. Specifically, the image data 202 may be stored in an external storage device, and the image analysis device 1 may read and use the image data 202 from the external storage device at necessary timing. Also, of course, the image analysis device 1 does not need to be an information processing device owned by an expert, and may be realized using cloud computing.

Processing executed in the image analysis device 1 according to the embodiment will be described using the flowchart shown in FIG. 6. Note that the order of the following steps is not limited, and processes that can be executed simultaneously may be executed at the same time, or the order of the steps may be changed to the extent possible.

First, the extraction unit 101 extracts image data 202 from a data group. Further, the extraction unit 101 stores the extracted image data 202 in the storage unit 20 (S01).

The classification unit 102 uses the trained model M to classify the class of each image data 202 extracted in step S01. Furthermore, the classification unit 102 generates class data 203 indicating the classification result, and stores it in the storage unit 20 (S02). For example, the classification unit 102 classifies the image data 202 into a "first character class" that includes handwritten characters, a "second character class" that includes characters other than handwritten characters, or an "image class" that does not include characters. do. For example, when using the trained model M that distinguishes between "third character class" and "fourth character class" for characters other than handwritten characters according to the priority as evidence, the classification unit 102 The character class is classified into a "third character class" or a "fourth character class."

The conversion unit 103 converts each image data associated with a specific class in the classification result obtained in step S02 into text data 204 using character recognition technology. Furthermore, the conversion unit 103 stores the converted text data 204 in the storage unit 20 (S03). For example, each image data classified into the "second character class" is converted into text data 204 using character recognition technology. For example, the conversion unit 103 converts each image data classified into the "fourth character class" into text data 204 using character recognition technology.

The analysis unit 104 analyzes the text data 204 obtained in step S03. Furthermore, the analysis unit 104 stores the analysis result data 205 in the storage unit 20 (S04). For example, the analysis unit 104 analyzes the quantity of a specific word and uses the result as the analysis result data 205.

The output processing unit 105 outputs the classification result obtained in step S02 and the analysis result obtained in step S04 for each data to the output unit 50 (S05).

As described above, according to the image analysis method, image analysis device, and image analysis program according to the present disclosure, by classifying the image data 202 by attribute, it is possible to easily determine subsequent steps. Or, if the analysis process has already been performed, the contents of the image data 202 can be grasped, which reduces the burden on experts in data investigation in forensics, shortens the required time, and improves the efficiency of the work. becomes possible.

<Modified example>
An image analysis device 1A according to a modification will be described using FIG. 7. Compared to the image analysis device 1 shown in FIG. 3, the image analysis device 1A according to the modified example differs in that character data 201 is stored in the storage unit 20. In addition, in the modified example, reference numerals are underlined for configurations and processes that are different from the above-described embodiment.

Character data 201 is data extracted from the data group. In the image analysis device 1A according to the modification, the extraction unit 101 extracts the character data 201 as well as the image data 202 from the data group. Furthermore, the analysis unit 104 analyzes the contents of the character data 201 as well as the text data 204. Thereby, in the image analysis device 1A according to the modification, both the character data 201 and the text data 204 generated from the image data 202 can be analyzed, and the analysis results can be treated equally. Therefore, the analysis result data 205 can include the result of character analysis of the character data 201.

For example, the analysis result data 205 includes the quantity of specific words included in each text data 204 and each character data 201 in association with the identification information of each text data 204 and each character data 201. Thereby, according to the image analysis device 1A, it becomes possible to select necessary information from the contents of the data, regardless of the type of data included in the data group, and it becomes possible to use it for review and document judgment. .

When handling such text data 204 and character data 201, it is possible to determine subsequent steps for each data type and class. For example, if the character information included in the image data 202 that corresponds to the first character class and the third character class is likely to have a meaning such as writing a memo, the first character class and the third character class The text data 204 obtained from the image data 202 that corresponds to the character class may be treated equally. Furthermore, if it is thought that the character information included in the image data 202 and the character information included in the character data 201 that correspond to the fourth character class are often meant to explain the content of the target project, Text data 204 generated from image data 202 of character class No. 4 and character data 201 may be treated equally.

Processing executed in the image analysis device 1A according to the modification will be described using the flowchart shown in FIG. 8. Note that the order of the following steps is not limited, and processes that can be executed simultaneously may be executed at the same time, or the order of the steps may be changed to the extent possible.

First, the extraction unit 101 extracts character data 201 and image data 202 from a data group, and stores them in the storage unit 20 (S11).

The classification unit 102 uses the learned model M to classify the classes of each image data 202 extracted in step S11, generates class data 203 indicating the classification result, and stores it in the storage unit 20 (S02). .

The conversion unit 103 converts each image data associated with a specific class in the classification result obtained in step S02 into text data 204 using character recognition technology, and converts the text data 204 obtained by the conversion into text data 204. The information is stored in the storage unit 20 (S03).

The analysis unit 104 analyzes the text data 204 obtained in step S03, and stores the analysis result data 205 in the storage unit 20 (S04).

Furthermore, the analysis unit 104 analyzes the character data 201 extracted in step S11, and adds the analysis result to the analysis result data 205 in the storage unit 20 (S15).

The output processing unit 105 outputs the classification results of step S02 and the classification results of step S04 and step S15 for each data to the output unit 50 (S16).

As described above, according to the image analysis method, image analysis device, and image analysis program according to the modified example, by classifying image data by attribute, it is possible to easily determine subsequent steps, Alternatively, if analysis processing has already been performed, the contents of the image data 202 can be grasped, and the analysis results of the text data 204 obtained from the image data 202 and the analysis results obtained from the character data 201 can be parallelized. This makes it possible to reduce the burden on experts in forensic data investigation, shorten the time required, and improve efficiency.

The present disclosure is useful for screening target image data when handling image data in digital forensics.

1 Image analysis device 10 Control unit 101 Extraction unit 102 Classification unit 103 Conversion unit 104 Analysis unit 105 Output processing unit 20 Storage unit 201 Character data 202 Image data 203 Class data 204 Text data 205 Analysis result data P Image analysis program M Learned model

Claims (20)

extracting image data from a data group including a plurality of character data and a plurality of image data;
obtaining the extracted image data;
A plurality of image data, and the class of each image data is either a first character class that includes handwritten characters, a second character class that includes characters other than handwritten characters, or an image class in which each image data does not include characters. classifying the class of the obtained image data using a trained model that has learned training data associated with the image data;
Image analysis methods including. The data group is forensic data used for a specific investigation,
The second character class includes a third character class that is a class of data with a high priority as evidence in forensics, and a class of data that has a lower priority as evidence than the data of the third character class. a fourth character class;
In the step of classifying the class of the image data, classifying the image data as one of the first character class, the third character class, the fourth character class, or the image class;
The image analysis method according to claim 1. The trained model is trained using teacher data to which the first character class is associated as a class of image data that includes at least both handwritten characters and non-handwritten characters. image analysis method. The learned model is a class of image data that includes at least both handwritten characters and non-handwritten characters, and the number of handwritten characters is a predetermined ratio or more to the number of non-handwritten characters. The image analysis method according to claim 1 or 2, wherein the image analysis method has been trained using teacher data with which character classes are associated. extracting image data from a data group that is forensic data used in a specific investigation and includes a plurality of character data and a plurality of image data;
obtaining the extracted image data;
A plurality of image data, and at least two classes of each image data: a class of data that includes characters that have a high priority as evidence in forensics, and a class of data that has a lower priority as evidence than that class. classifying the acquired image data using a trained model that has been trained on training data to which any one of the acquired image data is associated;
Image analysis methods including. storing class data indicating the classified class in a storage unit in association with each of the image data into which the class has been classified;
converting, in the storage unit, only the image data associated with a class containing characters with high priority as evidence into text data using a processing result of character recognition;
determining the quantity of specific words included in the text data;
The image analysis method according to claim 2, comprising: storing the determined quantity of the specific word in a storage unit in association with the image data that is the source of the text data;
The image analysis method according to claim 6, comprising: outputting the image data in association with a ranking according to the quantity of the specific words included in the text data;
The image analysis method according to claim 7, comprising: 9. The image analysis method according to claim 8, wherein in the step of outputting the image data, an attribute indicating the associated class is also output for each of the output image data. The image analysis method according to claim 8, wherein in the step of outputting the image data, the class associated with the image data is selected as a filter and displayed on a display. The image analysis method according to claim 8, wherein in the step of outputting the image data, only the image data associated with the third character class is output. The image analysis method according to claim 8, wherein in the step of outputting the image data, only the image data associated with the third character class and the fourth character class are output. The image analysis method according to claim 8, wherein in the step of outputting the image data, the image data associated with the third character class is output with priority over other image data. In the step of outputting the image data, the image data associated with the third character class, the image data associated with the fourth character class, and the image data associated with the first character class. , the image analysis method according to claim 8, wherein the images are output in the order associated with the image class. The image analysis method according to claim 8, wherein in the step of outputting the image data, the image data associated with the third character class is output with priority over the character data. calculating the quantity of the specific words included in the plurality of document data of the data group;
The text data in association with the calculated quantity of the specific words included in the plurality of document data and a ranking according to the quantity of the specific words included in the text data stored in the storage unit. outputting the image data and the document data that are the basis of;
The image analysis method according to claim 7, comprising: The image analysis method according to claim 7, wherein the quantity of the specific words is either the number of the specific words included in the data or the proportion of the specific words in the number of words in the entire data. 8. The image analysis method according to claim 7, wherein in the step of outputting, the file name of the image data and the attribute of the class into which the image data is classified are displayed in association with each other. A computer program that causes a computer to execute the image analysis method according to claim 1. an extraction unit that extracts image data from a data group including a plurality of document data and a plurality of image data;
A plurality of image data, and the class of each image data is either a first character class that includes handwritten characters, a second character class that includes characters other than handwritten characters, or an image class in which each image data does not include characters. a classification unit that classifies the extracted image data using a trained model that has trained training data associated with the image data;
Image analysis equipment including.