JP7843902B1 - Image analysis device, information processing method, and program - Google Patents

Abstract

[Problem] To provide a technology that improves the accuracy of analysis of the imaged object.
[Solution] An image analysis device according to one aspect of the present disclosure includes a control unit configured to acquire an image captured after being given instructions for the imaging composition, to provide the acquired image and instructions for the imaging composition to an analysis model, to have the analysis model perform image analysis, to acquire the image analysis results from the analysis model, and to output information regarding the analysis results of the acquired image.
[Selection Diagram] Figure 17

Description

This disclosure relates to an image analysis device, an information processing method, and a program.

Non-Patent Document 1 proposes a system for understanding documents using a large-scale language model. In Non-Patent Document 1, visual information such as diagrams and charts contained within document images is made to be understood by the large-scale language model along with the text.

NTT, “Establishing Visual Reading Comprehension Technology Using LLM - Towards the Realization of ‘tsuzumi’ for Understanding Graphical Documents,” [online], [Retrieved October 31, 2025], Internet <URL: https://group.ntt/jp/newsrelease/2024/04/12/240412b.html?utm_source=chatgpt.com>

In recent years, the development of large-scale (visual) language models has progressed, making it easier to perform image analysis. For example, conventional methods, such as those described in Non-Patent Document 1, allow for image analysis by providing an image with a predetermined configuration to the analysis model. However, the inventors of this invention have found the following problems with conventional methods: Specifically, conventional methods fail to provide the analysis model with information that serves as a hint for analysis, such as the object of interest or how the object is depicted. Misidentification of this information can lead to a decrease in the accuracy of the analysis performed by the analysis model.

In one respect, this disclosure was made in consideration of these circumstances. One of the purposes of this disclosure is to provide a technology that improves the accuracy of the analysis of the imaged object.

This disclosure adopts the following configuration to solve the aforementioned problems. Note that the following configurations can be combined as appropriate.

An image analysis device relating to one aspect of this disclosure includes a control unit configured to acquire an image captured after being given instructions for the imaging composition, provide the acquired image and the instructions for the imaging composition to an analysis model, have the analysis model perform the image analysis, obtain the image analysis results from the analysis model, and output information related to the analysis results of the acquired image. According to this configuration, in addition to the acquired image, instructions for the imaging composition used when taking the image are also provided to the analysis model. This allows the analysis model to be given information that serves as a hint for the analysis. Therefore, according to this configuration, an improvement in the accuracy of the analysis of the imaged object can be expected.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requiring the placement of a predetermined location of the object to be imaged according to a marker on the imaging screen. This configuration allows for improved accuracy in the analysis of the object by clearly defining its position.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requiring the addition of a physical marker to a predetermined location on the object to be imaged. With this configuration, by clarifying the position of the object to be imaged, it is possible to expect an improvement in the accuracy of the analysis of the object to be imaged.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requiring that a predetermined area of the object to be imaged be covered. With this configuration, by covering areas other than the object to be imaged, an improvement in the accuracy of the analysis of the object to be imaged can be expected.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that a predetermined part of the object to be imaged be placed at a predetermined position in the image. This configuration allows for improved accuracy in the analysis of the object by clearly defining its position.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that the object to be imaged be imaged in a predetermined orientation. According to this configuration, by clearly defining the orientation of the object to be imaged, an improvement in the accuracy of the analysis of the object can be expected.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that the object to be imaged be imaged with a predetermined background color. This configuration allows for a clear distinction between the object to be imaged and other areas, thereby improving the accuracy of the analysis of the object to be imaged.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition may be an instruction requesting that the object to be imaged be imaged under predetermined illumination conditions. This configuration allows for improved accuracy in the analysis of the object by enhancing its visibility.

In an image analysis device relating to one aspect of this disclosure, the imaging composition instruction may be specified at the time of image acquisition. This allows for the output of different imaging composition instructions for each image. If the imaging composition instruction differs for each image, it becomes necessary to change the composition of the disguised object for each instruction, which is time-consuming. Therefore, this configuration can be expected to suppress the disguise of the imaged object.

In an image analysis device relating to one aspect of this disclosure, specifying the imaging composition during image acquisition may be configured by selecting one or more instruction candidates from a plurality of instruction candidates and designating the selected one or more instruction candidates as the imaging composition instruction. This allows for the output of different imaging composition instructions for each image acquisition. Therefore, this configuration can be expected to suppress the falsification of the image target.

In an image analysis device relating to one aspect of this disclosure, the instruction for the imaging composition includes one or more parameters, and specifying the instruction for the imaging composition when capturing an image may include specifying the values of each of the one or more parameters. This allows for the output of different imaging composition instructions for each image capture. This configuration is expected to suppress the falsification of the image target.

In an image analysis device relating to one aspect of this disclosure, the image contains a document as the object to be captured, and the image analysis may include identifying at least a portion of the document's content. This configuration allows for improved accuracy in the analysis of the object to be captured when analyzing a document.

In an image analysis device relating to one aspect of this disclosure, the image contains an object as the object to be captured, and the image analysis may include evaluating at least one of the condition and authenticity of the object. According to this configuration, an improvement in the accuracy of the analysis of the object to be captured can be expected when evaluating the condition or authenticity of the object.

In an image analysis device relating to one aspect of this disclosure, the image analysis may include identifying the object being imaged. With this configuration, an improvement in the accuracy of the analysis of the object being imaged can be expected when identifying the object being imaged.

Furthermore, the form of this disclosure is not limited to the image analysis apparatus described above. As another aspect of the image analysis apparatus relating to each of the above aspects, one aspect of this disclosure may be an image analysis method (control method) that implements all or part of the above configurations, a program, or a machine-readable storage medium that stores such a program. Here, a machine-readable storage medium may be a non-temporary medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical action. Non-temporary storage media may include storage media (CDs, DVDs, semiconductor memory, etc.), auxiliary storage devices of computers, external storage devices connected to computers, etc.

For example, an image analysis method relating to one aspect of this disclosure may be performed by a computer. The image analysis method may include acquiring an image captured with instructions for the imaging composition, providing the acquired image and instructions for the imaging composition to an analysis model, having the analysis model perform the image analysis, obtaining the image analysis results from the analysis model, and outputting information regarding the analysis results of the acquired image.

For example, a program relating to one aspect of this disclosure may be a program that causes a computer to execute an image analysis method. The image analysis method may include acquiring an image captured with instructions for the imaging composition, providing the acquired image and instructions for the imaging composition to an analysis model, causing the analysis model to perform image analysis, obtaining the image analysis results from the analysis model, and outputting information regarding the analysis results of the acquired image.

According to one aspect of this disclosure, the accuracy of the analysis of the imaged object can be improved.

Figure 1 is a schematic diagram illustrating an example of a scenario to which this disclosure applies. Figure 2 schematically shows an example of a marker on the imaging screen. Figure 3 schematically shows an example of a marker on the imaging screen. Figure 4 schematically shows an example of a marker on the imaging screen. Figure 5 schematically shows an example of a scenario in which physical markers are added. Figure 6 schematically shows an example of a scene in which a predetermined area of the imaging target is covered. Figure 7 schematically shows an example of a scenario in which a predetermined part of the image to be captured is positioned at a predetermined location in the image. Figure 8 schematically shows an example of a scenario in which an object to be imaged is captured in a specified orientation. Figure 9 schematically shows an example of a scenario in which the object to be imaged is captured with a predetermined background color. Figure 10 schematically shows an example of a scene in which an object to be imaged is imaged under predetermined illumination conditions. Figure 11 schematically shows an example of a scene in which instructions for the imaging composition are given. Figure 12 schematically shows an example of a scene in which captured images are analyzed. Figure 13 schematically shows an example of a scene in which captured images are analyzed. Figure 14 schematically shows an example of a scene in which captured images are analyzed. Figure 15 schematically shows an example of the hardware configuration of an image analysis device. Figure 16 schematically shows an example of the software configuration of an image analysis device. Figure 17 is a flowchart showing an example of the processing procedure of this disclosure. Figure 18 shows the prompts entered in the comparative example. Figure 19 shows the prompts entered in the example. Figure 20 shows an example of a sample image input in the embodiment. Figure 21 shows the extraction results for the comparative example and the example.

The following describes embodiments relating to one aspect of this disclosure, based on the drawings. However, the embodiments described below are merely illustrative in all respects of this disclosure. Various improvements or modifications may be made without departing from the scope of this disclosure. In implementing this disclosure, specific configurations may be adopted as appropriate depending on the embodiment. While the data appearing in these embodiments is described in natural language, more specifically, it is specified using pseudo-language, commands, parameters, machine code, electrical signals, etc., that can be recognized by machines such as computers.

§1 Application Examples Figure 1 is a schematic diagram illustrating an example of a scenario to which this disclosure applies. The image analysis device 1 according to this embodiment is one or more computers configured to output information 60 regarding the analysis result 50 of an image (captured image 30) captured in response to an instruction 20 for the imaging composition. In one example, the image analysis device 1 may be a server device. In another example, the image analysis device 1 may be a user terminal.

In this embodiment, the image analysis device 1 provides user U with an instruction 20 for the imaging composition. User U obtains an image 30 by imaging the target IT with the imaging device C according to the instruction 20 for the imaging composition. The image analysis device 1 acquires the image 30 from user U. The image analysis device 1 provides the instruction 20 for the imaging composition and the image 30 to the analysis model 40, causing the analysis model 40 to perform analysis of the image 30. The image analysis device 1 acquires the analysis result 50 of the image 30. Then, the image analysis device 1 outputs information 60 related to the acquired analysis result 50 of the image 30.

According to this embodiment, the image analysis device 1 provides the analysis model 40 with not only the captured image 30 but also instructions for the imaging composition 20. This allows the analysis model 40 to receive information that serves as a hint for the analysis. As a result, an improvement in the accuracy of the analysis of the imaged object can be expected.

[Target of imaging]
The image target IT may be any object that can be analyzed by the analysis model 40, such as a document, a person, or an object.

[Instructions for the composition of the image]
The imaging composition indicates how the target IT will be displayed in the image. Therefore, the imaging composition instruction 20 may be an instruction given to the user U in the scene of imaging the target IT, specifying how the target IT will be displayed in the image. The imaging composition instruction 20 is not particularly limited and may be set as appropriate, as long as the user U can image the target IT according to the instruction. For example, the imaging composition instruction 20 may be configured to include information that can serve as a hint for the analysis of the target IT by the analysis model 40. The imaging composition instruction 20 may also be configured to be understandable to the user U in natural language.

The imaging composition instruction 20 may include an instruction to limit the range to be analyzed by the analysis model 40. This improves the accuracy of the analysis of the captured image 30 by the analysis model 40.

In one example, the imaging composition instruction 20 may be an instruction requesting that a predetermined location of the imaging target IT be positioned according to a marker on the imaging screen. The marker on the imaging screen may be configured to be displayed superimposed on the imaging target IT. The shape, size, etc., of the marker displayed on the imaging screen are not particularly limited and may be appropriately defined depending on the embodiment. The marker may also be fixed and displayed at any position on the imaging screen. The predetermined location of the imaging target IT may be appropriately set depending on the type of imaging target IT and the purpose of the analysis.

The designated area may be the entirety of the IT being imaged, or a part of it. For example, if the designated area is a part of the IT being imaged, it may be an area that is heavily analyzed by the analysis model 40. For instance, if the IT being imaged is a document, the designated area may be a specified description within the document. Note that the designated area of a document may include descriptions on the back. Furthermore, if the IT being imaged is an item, the designated area may be a part that is particularly important in the evaluation of the item. For example, an important area may include a part of the item that is prone to damage (such as the corner of a card). Furthermore, if the IT being imaged is a person, the designated area may be a part that is effective for person identification. Typically, the designated area for a person may be the face.

As a result of imaging, the markers on the imaging screen may or may not be displayed in the captured image 30. If the markers are not displayed, the captured image 30 may be configured to retain information indicating the position of the markers. Accordingly, the analysis model 40 may be provided with information indicating the position of the markers in addition to the imaging composition instruction 20.

Figures 2 to 4 schematically show examples of markers (SM1, SM2, SM3) on the imaging screen IS. A receipt is selected as the imaging target IT1, and the scenario assumes an image is being captured to analyze the contents of the receipt. In this example, the "TOTAL" section is set as the predetermined location PL. In one example, the marker may be a marker SM1 indicating a point on the imaging screen IS, as shown in Figure 2. In this case, for example, the user U may be presented with a statement such as "Please align the marker on the screen with 'TOTAL' on the receipt and take the picture" as the imaging composition instruction 20. In another example, the marker may be straight lines SM2 positioned at both the left and right ends of the imaging screen IS, as shown in Figure 3. In this case, for example, the user U may be presented with a statement such as "Please align the height of the lines on the screen with the position of 'TOTAL' and take the picture" as the imaging composition instruction 20. In another example, the landmark may be a straight line SM3 connecting the left and right ends of the imaging screen IS, as shown in Figure 4. In this case, for example, the user U may be presented with a sentence such as "Please take the image so that 'TOTAL' is positioned below the line on the screen" as the imaging composition instruction 20.

For example, the imaging composition instruction 20 may be an instruction requesting the addition of a physical marker to a predetermined location on the target IT. For instance, adding a physical marker may involve placing any object. The imaging composition instruction 20 may include an instruction specifying a particular object as the physical marker. In this case, the object serving as the physical marker can be arbitrarily selected (e.g., a pen, fingertip) that does not affect the analysis of the target IT. Alternatively, the imaging composition instruction 20 does not necessarily have to include an instruction specifying a particular object as the physical marker. In this case, the image analysis device 1 (analysis model 40) may identify the physical marker by extracting the object arbitrarily selected and placed by the user U from the captured image 30. The predetermined location on the target IT may be set appropriately according to the type of target IT and the purpose of the analysis, similar to the case where markers are displayed on the imaging screen.

Figure 5 schematically shows an example of a scenario in which a physical marker is added. A receipt is selected as the image target IT1, and the scenario assumes that a marker is placed when capturing an image to analyze the contents written on the receipt. In this example, the part with "TOTAL" written on it is set as the predetermined location PL. In one example, the marker may be a pen PM, as shown in Figure 5. In this case, for example, the user U may be presented with a sentence such as "Align the pen tip with the part corresponding to 'TOTAL' and take a picture" as the image composition instruction 20. The above case is when a physical marker is specified. Otherwise, for example, the user U may be presented with a sentence such as "Place the marker on the part corresponding to 'TOTAL' and take a picture" as the image composition instruction 20.

Furthermore, adding physical markers may involve applying predetermined processing to the imaging target IT, provided that it does not affect the analysis. For example, adding physical markers may include forming markers on the imaging target IT by writing.

For example, the imaging composition instruction 20 may be an instruction requesting that a predetermined area of the imaging target IT be covered. For instance, covering may be achieved by placing an object to obscure at least a portion of the predetermined area. The imaging composition instruction 20 may include an instruction specifying a particular object as the covering. In this case, the covering object could be any object, such as a hand or a sheet of paper. Alternatively, the imaging composition instruction 20 may not include an instruction specifying a particular object as the covering. In this case, the image analysis device 1 (analysis model 40) may identify the covering object by extracting the object arbitrarily selected and placed by the user U from the captured image 30. The predetermined area for covering may be appropriately set depending on the type of imaging target IT and the purpose of the analysis. For example, the predetermined area for covering could be any part that is not the target of analysis.

Furthermore, covering may include covering unwanted portions by applying a predetermined process to the captured image 30. For example, the predetermined process may include filling in, mosaic processing, blurring, etc., of unwanted portions in the captured image 30.

Figure 6 schematically shows an example of a scenario where a predetermined area UP of the image target IT is covered. The image target IT1 is a receipt, and the scenario assumes the need to cover unnecessary parts when capturing an image for analyzing the contents of the receipt. In this example, the predetermined area UP may be any part that is not the target of analysis. For example, when analyzing the "TOTAL" section, the user U may be presented with an instruction 20 for the imaging composition such as, "Please cover everything except the part corresponding to 'TOTAL' when taking the image." Alternatively, without explicitly specifying the target of analysis, the user U may be presented with an instruction 20 for the imaging composition such as, "Please cover any irrelevant or unnecessary information when taking the image."

In one example, the imaging composition instruction 20 may be an instruction requesting that a predetermined location of the target IT be placed at a predetermined position on the image (imaging image 30). The predetermined location can be arbitrarily set to the top/bottom, center, top/bottom edge of the image, etc. The predetermined location may also be the entire image. For example, if the imaging image 30 is captured as a magnified image of a predetermined location of the target IT, and as a result the predetermined location of the target IT is captured across the entire image, then the predetermined location may be the entire image. The predetermined location of the target IT may be set appropriately according to the type of target IT and the purpose of analysis, similar to how markers are displayed on the imaging screen.

Figure 7 schematically illustrates an example of positioning a predetermined location of the image target IT at a predetermined position in the image. A receipt is selected as the image target IT1, and the scenario assumes capturing an image for analyzing the contents of the receipt. In this example, the top edge of the receipt is set as the predetermined location. Furthermore, the predetermined location is set as the top edge of the image (image capture screen IS). In this case, for example, the user U may be presented with a message such as "Please align the top edge of the receipt with the top edge of the image before taking the picture" as an instruction 20 for the image composition.

Furthermore, the imaging composition instruction 20 may include instructions to improve the identifiability of the object to be analyzed. This can improve the accuracy of the analysis of the captured image 30 by the analysis model 40.

In one example, the imaging composition instruction 20 may be an instruction requesting that the imaging target IT be imaged in a predetermined orientation. The predetermined orientation may be specified as appropriate depending on the imaging target IT. For example, if the imaging target IT is a document, the predetermined orientation may be specified so that the items of the document are in the correct orientation. Also, if the imaging target IT is an object, an orientation appropriate to the object, such as the front, side, or top, may be specified.

Figure 8 schematically illustrates an example of a scenario in which an image target IT is captured in a specified orientation. The image target IT2 is a receipt containing multiple orientations of text, and the scenario assumes the capture of an image for analyzing the contents of the receipt. In this example, the predetermined orientation is set so that the portion containing "Receipt" is upright (correctly oriented). In this case, for example, the user U may be presented with a message such as "Please take the image so that the portion corresponding to 'Receipt' is in the correct orientation" as the image composition instruction 20.

In one example, the imaging composition instruction 20 may be an instruction requesting that the imaging target IT be imaged with a predetermined background color. The predetermined background color may be any background color that is distinguishable from the imaging target IT. For example, the predetermined background color may be a single color (or a color close to a single color).

Figure 9 schematically illustrates an example of a scenario in which an image target IT is captured against a predetermined background color BG. A trading card is selected as the image target IT3, and the scenario assumes the capture of an image for analysis of the trading card. In this example, black is specified as the predetermined background color BG. In this case, for example, a statement such as "Please take the image against a black background" may be presented to the user U as an instruction 20 regarding the imaging composition.

For example, the imaging composition instruction 20 may be an instruction requesting that the imaging target IT be imaged under predetermined lighting conditions. These predetermined lighting conditions may include requirements related to the image appearance, such as illumination from a specified angle, suppression of overexposure, and the addition of shadows.

Figure 10 schematically illustrates an example of imaging an object (IT) under predetermined lighting conditions. A trading card is selected as the object (IT3), and the scenario assumes the acquisition of an image for analysis of the trading card. In this example, the predetermined lighting condition specifies illumination from the right. In this case, for example, a statement such as "Please illuminate from the right and take the image" may be presented to the user (U) as an instruction 20 regarding the imaging composition.

(How to specify instructions)
The imaging composition instruction 20 may be specified at any time. In one example, the imaging composition instruction 20 may be a fixed instruction specified in advance. In this case, the specified imaging composition instruction 20 may be stored in the memory resources of the image analysis device 1. In another example, the imaging composition instruction 20 may be specified when the image 30 is captured. That is, the imaging composition instruction 20 may be specified when user U starts the imaging device C.

Figure 11 schematically shows an example of a scenario in which the imaging composition instruction 20 is specified. In one example, specifying the imaging composition instruction 20 when capturing the image 30 may be done by selecting one or more instruction candidates from a plurality of instruction candidates 200, and then specifying the selected instruction candidate as the imaging composition instruction 20. The instruction candidates 200 may be predetermined instructions corresponding to the analysis content and may be stored in the memory resources of the image analysis device 1. The method for selecting the instruction to be the imaging composition instruction 20 from the instruction candidates 200 is not particularly limited and may be defined as appropriate. For example, the selection method may be a random selection method from the instruction candidates 200, a sequential selection method, etc.

The imaging composition instruction 20 may be configured to include one or more parameters 250. Accordingly, specifying the imaging composition instruction 20 when capturing the image 30 may be done by specifying the value of each of the one or more parameters 250. The parameters 250 may be predetermined according to the content of the imaging composition instruction 20. For example, the parameters 250 may include the position of the marker on the imaging screen IS, the position of the imaging target IT, the background color BG, etc. The method for specifying the parameters 250 may be set as appropriate, such as a method of probabilistically determining the parameter value, or a method of selecting from multiple parameter candidates.

Furthermore, the method for specifying the imaging composition instruction 20 may be a combination of the above-described methods. That is, the instruction candidates 200 may include instruction candidates that include the parameter 250. Accordingly, specifying the imaging composition instruction 20 may consist of selecting one or more instruction candidates from among the multiple instruction candidates 200, and specifying the value of the parameter 250 included in the selected instruction candidate.

(How to give instructions to the user regarding the image composition)
The imaging composition instruction 20 may be given to the user U from the image analysis device 1, or it may be given to the user U via another device. If it is given via another device, the image analysis device 1 may obtain the imaging composition instruction 20 from that other device. For example, the image analysis device 1 may be a server device and the other device may be a user terminal, in which case the image analysis device 1 may obtain the imaging composition instruction 20 from the user terminal.

The imaging composition instruction 20 may be presented to the user U at any time before imaging. In one example, the imaging composition instruction 20 may be presented before the imaging device C is started. In another example, the imaging composition instruction 20 may be presented simultaneously with the start of the imaging device C.

[Analysis Model]
The configuration of the analysis model 40 is not particularly limited and may be determined as appropriate depending on the embodiment, as long as it has the ability to perform analysis based on the given imaging composition instruction 20 and the captured image 30. In one example, the analysis model 40 may be composed of a trained machine learning model. The configuration (type, structure, etc.) of the machine learning model may be determined arbitrarily. The trained machine learning model may include a large-scale generative model such as a large-scale visual language model (VLM). The large-scale generative model used in the analysis model 40 may be a multimodal model.

The analysis model 40 may be stored in the memory resources of the image analysis device 1, or it may be stored in an external computer. If the analysis model 40 is stored in an external computer, the image analysis device 1 may transmit the imaging composition instruction 20 and the captured image 30 to the external computer via a network and obtain the analysis result 50.

Figures 12 to 14 schematically show an example of analysis performed by the analysis model 40. The content of the analysis performed by the analysis model 40 is not particularly limited, as long as it obtains any information related to the imaging target IT, and may be appropriately selected depending on the embodiment. In this case, the analysis model 40 may be configured as a dedicated model trained to perform a specific analysis.

As shown in Figure 12, the imaged IT may be a document. That is, the captured image 30 may contain a document. Accordingly, the analysis of the captured image 30 by the analysis model 40 may include identifying at least a portion of the document's content. In this case, the analysis model 40 may be trained to identify the document's content. The document may be any document containing text. For example, identifying the document's content may include content identification such as receipt analysis or ingredient label analysis.

As shown in Figure 13, the object to be imaged (IT) may be an object. That is, the imaged image 30
The image may contain an object. Accordingly, the analysis of the captured image 30 by the analysis model 40 may include evaluating at least one of the condition and authenticity of the object. In this case, the analysis model 40 may be trained to evaluate the condition/authenticity of the object. The object may be a tangible object that can be the subject of evaluation. For example, the object may be trading cards, branded goods, clothing, etc., which can be evaluated for condition/authenticity at the time of transaction. The object may also be infrastructure equipment, defective goods, etc., which can be evaluated for condition. Condition evaluation may consist of estimating the condition of the object. Specifically, condition evaluation may consist of estimating the degree of damage, dirt, breakage, defects, deformation, wear, etc. of the object. Authenticity evaluation may consist of determining whether the object is genuine or not.

As shown in Figure 14, the imaged object (IT) may be any object, including people and objects. Accordingly, the analysis of the captured image 30 by the analysis model 40 may include identifying the imaged object (IT). In this case, the analysis model 40 may be pre-trained to identify the imaged object (IT). For example, identification may include identity verification, person identification, category determination, attribute determination, etc.

(How to provide instructions for the imaging composition to the analysis model)
The imaging composition instruction 20 given to the analysis model 40 may or may not be the same as the one given to the user U. That is, the imaging composition instruction 20 may be appropriately converted to indicate its meaning before being given to the analysis model 40. In one example, an information statement indicating the content of the imaging composition instruction 20 may be generated by manual input by an operator of the image analysis device 1. In another example, an information statement indicating the content of the imaging composition instruction 20 may be generated according to a pre-prepared template. The image analysis device 1 may provide the analysis model 40 with the information statement. The format of the above information statement is not particularly limited and may be defined as appropriate.

The image analysis device 1 may determine whether the captured image 30 was captured according to the imaging composition instruction 20 after acquiring the captured image 30, before providing the captured image 30 to the analysis model 40. The image analysis device 1 may perform this determination using an identification model separate from the analysis model 40. The identification model may be stored in the memory resources of the image analysis device 1 or on an external computer. If the identification model is stored on an external computer, the image analysis device 1 may transmit the imaging composition instruction 20 and the captured image 30 to the external computer via a network and have the determination performed. The image analysis device 1 may obtain the determination result from the external computer. If it is determined that the image was not captured according to the imaging composition instruction 20, the image analysis device 1 may provide the imaging composition instruction 20 to the user U again and obtain the captured image 30. The image analysis device 1 may provide the imaging composition instruction 20 to the user U and instruct them to capture until it is determined that the image was captured according to the imaging composition instruction 20.

The image analysis device 1 obtains the analysis result 50 by providing the imaging composition instruction 20 and the captured image 30 to the analysis model 40. The information provided to the analysis model 40 may include information other than the imaging composition instruction 20 and the captured image 30. For example, the image analysis device 1 may provide the analysis model 40 with instructions regarding the output format of the analysis result 50. In one example, the image analysis device 1 may provide instructions to output the analysis result 50 in natural language. In another example, the image analysis device 1 may provide instructions to output the analysis result 50 in a predetermined file format (such as JSON).

[Output information regarding the analysis results]
In a typical example, outputting information 60 related to the analysis result 50 may be done by outputting the analysis result 50 as is. However, the content of information 60 is not limited to this example, as long as it relates to the analysis result 50, and may be determined appropriately depending on the embodiment. In another example, the image analysis device 1 may perform arbitrary information processing on the analysis result 50. Outputting information 60 related to the analysis result 50 may be done by outputting the result of this information processing together with the analysis result 50 or in place of the analysis result 50.

§2 Configuration Examples [Hardware Configuration]
Figure 15 schematically shows an example of the hardware configuration of the image analysis device 1 according to this embodiment. In one example, the image analysis device 1 may be configured as a computer in which a control unit 11, a storage unit 12, an external interface 13, a communication interface 14, an input device 15, and an output device 16 are electrically connected.

The control unit 11 is configured to perform information processing based on the program and various data. For example, the control unit 11 may include a hardware processor such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). The control unit 11 (CPU) is an example of a processor resource.

The storage unit 12 is configured to hold arbitrary data. For example, the storage unit 12 may include a hard disk drive, a solid-state drive, semiconductor memory, etc. The storage unit, RAM, and ROM are examples of memory resources. In one example, the storage unit 12 may store a program 81. Program 81 is a program that causes the image analysis device 1 to execute the information processing (described later in Figure 17) according to this disclosure. Program 81 includes a series of instructions for said information processing.

In one example, program 81 may be stored in the storage medium 91 instead of, or together with, the storage unit 12. The storage medium 91 is configured to store various types of information (stored programs, etc.) through electrical, magnetic, optical, mechanical, or chemical means so that a machine such as a computer can read the information. The storage unit 12 and the storage medium 91 are examples of non-temporary storage media. The image analysis device 1 may retrieve program 81 from the storage medium 91. The storage medium 91 may be a disk-type storage medium (CD, DVD, etc.) or a non-disk-type storage medium such as semiconductor memory (flash memory, etc.). Any drive device may be used to read the information stored in the storage medium 91. The type of drive device may be selected according to the storage medium 91. The drive device may be connected to the image analysis device 1 in any way. The storage medium 91 may include an external storage device that can be connected to the image analysis device 1.

The external interface 13 may be configured to connect to an external device via wired or wireless connection, for example, by a USB (Universal Serial Bus) port, a dedicated port, etc. The communication interface 14 is configured to perform wired or wireless communication over a network. The communication interface 14 may be configured by, for example, a wired LAN (Local Area Network) module, a wireless LAN module, etc. The network standard is not particularly limited and may be appropriately selected depending on the embodiment. For example, the network type may be appropriately selected from the Internet, wireless communication network, mobile communication network, telephone network, dedicated network, etc. The image analysis device 1 may perform data communication with other computers via the communication interface 14.

The input device 15 is configured to accept information input. The input device 15 may consist of, for example, a camera, microphone, mouse, keyboard, touch panel, or operator. The output device 16 is configured to output information. The output device 16 may consist of, for example, a display or speaker. The image analysis device 1 may be operated using the input device 15 and the output device 16. The input device 15 and the output device 16 may be directly connected to the image analysis device 1, or they may be indirectly connected via at least one of the external interface 13 and the communication interface 14. The input device 15 and the output device 16 may be integrated at least in part by a touch panel display or the like.

Regarding the specific hardware configuration of the image analysis device 1, components can be omitted, replaced, and added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. Hardware processors include microprocessors, FPGAs (Field-Programmable Gate Arrays), DSPs (Digital Signal Processors), and ECUs.
It may consist of an Electronic Control Unit (ECC), a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), etc. The program 81 may be stored in an external storage device such as a Network Attached Storage (NAS). An external storage device is also an example of a non-temporary storage medium.

[Software Configuration]
Figure 16 schematically shows an example of the software configuration of the image analysis device 1 according to this embodiment. The control unit 11 of the image analysis device 1 executes instructions contained in the program 81 stored in the storage unit 12 using the CPU. As a result, the image analysis device 1 operates as a computer equipped with an instruction provision unit 111, an image acquisition unit 112, an analysis processing unit 113, and an output processing unit 114 as software modules. In other words, in one example, each software module of the image analysis device 1 may be implemented by the control unit 11 (CPU).

The instruction provision unit 111 is configured to generate an instruction 20 for the imaging composition and provide it to the user U. The image acquisition unit 112 is configured to acquire the image 30 captured by the user U. The analysis processing unit 113 is configured to provide the imaging composition instruction 20 and the image 30 to the analysis model 40 and acquire the analysis result 50. The output processing unit 114 is configured to output information 60 related to the analysis result 50.

In this example of the embodiment described above, each software module of the image analysis device 1 is implemented using a general-purpose CPU. However, the method of implementing each of the above modules is not limited to this example and may be appropriately modified depending on the embodiment. Some or all of the above software modules may be implemented using one or more dedicated processors or chipsets. Each of the above modules may also be implemented as a hardware module. Regarding the software configuration of the image analysis device 1, modules may be omitted, replaced, or added as appropriate, depending on the embodiment.

§3 Operation Example Figure 17 is a flowchart showing an example of a processing procedure by the image analysis device 1 according to this embodiment. The control unit 11 of the image analysis device 1 executes instructions contained in the program 81 held in the storage unit 12. As a result, the image analysis device 1 operates as a computer capable of performing the following information processing. The control unit 11 is configured to perform the following information processing. The following processing procedure is an example of an information processing method performed by a computer. However, the following processing procedure is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure can be omitted, replaced, and added as appropriate, depending on the embodiment.

(Step S101)
In step S101, the control unit 11 operates as an instruction providing unit 111 and provides the user U with an instruction 20 for the imaging composition. In one example of analysis, the instruction 20 for the imaging composition may be a fixed instruction specified in advance. In another example, the instruction 20 for the imaging composition may be specified when the user U takes the image 30. Once the instruction 20 for the imaging composition is given to the user U, the control unit 11 proceeds to the next step 102.

(Step S102)
In step S102, the control unit 11 operates as an image acquisition unit 112 and acquires the captured image 30 taken by user U. Once the captured image 30 is acquired, the control unit 11 proceeds to the next step S103.

Furthermore, after acquiring the captured image 30, the control unit 11 may determine whether the captured image 30 was captured according to the imaging composition instruction 20. This determination may be performed using an arbitrary identification model. The identification model may be stored in the storage unit 12 or in an external computer. If it is determined that the image was not captured according to the imaging composition instruction 20, the control unit 11 may return to step S101, re-issue the imaging composition instruction 20 to the user U, and instruct the system to capture the image. The control unit 11 may repeatedly execute steps S101 and S102 until it is determined that the captured image 30 was captured according to the imaging composition instruction 20.

(Step S103)
In step S103, the control unit 11 operates as an analysis processing unit 113, providing the imaging composition instruction 20 and the captured image 30 to the analysis model 40, and obtaining the analysis result 50. The analysis model 40 may be composed of a trained machine learning model. The analysis model 40 may be stored in the storage unit 12 or on an external computer. If the analysis model 40 is stored on an external computer, the control unit 11 may transmit the imaging composition instruction 20 and the captured image 30 to the external computer via the network and obtain the analysis result 50. The imaging composition instruction 20 given to the analysis model 40 may or may not be the same as the one given to user U. Once the analysis result 50 is obtained, the control unit 11 proceeds to the next step S104.

(Step S104)
In step S104, the control unit 11 operates as an output processing unit 114 and outputs information 60 related to the analysis result 50. In one example, the control unit 11 may output the analysis result 50 as is. In another example, the control unit 11 may perform arbitrary information processing on the analysis result 50. The control unit 11 may output the result of this information processing together with the analysis result 50 or in place of the analysis result 50.

The output destination of the output information 60 is not particularly limited and may be appropriately selected depending on the embodiment. The output destination may be, for example, RAM, storage unit 12, output device 16, storage medium 91, external computer, external storage device, etc. Once the output of information 60 is complete, the control unit 11 terminates the processing procedure according to this operation example.

§4 Modifications Although embodiments of the present disclosure have been described in detail above, the above description is merely illustrative in all respects of the present disclosure. Needless to say, various improvements or modifications can be made without departing from the scope of the present disclosure. The processes and means described in the present disclosure can be freely combined and implemented as long as no technical inconsistencies arise.

§5 Experimental Examples In order to verify that the accuracy of the analysis of the imaging target (analysis target) can be improved by providing instructions for the imaging composition to the analysis model, the following experiment was conducted. However, this disclosure is not limited to the following experimental examples.

First, the analysis will focus on 973 SROIE (Scanned Receipts OCR and Information) entries.
A dataset was prepared (Extraction). The task of extracting company names from receipt images in the above dataset was performed on the analysis model using the comparative example (standard input method) and the example (input method based on this disclosure) input methods, respectively. The results of extracting company names using the above two methods were obtained. The analysis model used was GPT-4o (OpenAI).

Figure 18 shows the prompts entered in the comparative example. In the comparative example, the input was an instruction to extract a receipt image and the company name. Figure 19 shows the prompts entered in the embodiment. In the embodiment, in addition to the instruction to extract a receipt image and the company name, contextual information of the instruction (information regarding the imaging composition) was provided as input. Figure 20 shows an example of a sample image entered in the embodiment. As shown in Figure 20, a cross marker was placed near the location where the company name is written in the receipt image. The contextual information for the embodiment stated that a visual marker was placed near the company name.

Figure 21 shows the extraction results for the comparative example and the example. The extraction results were classified into exact match, partial match, and no match. An exact match was defined as the extracted string exactly matching the correct company name. A partial match was defined as the extracted string matching the correct company name to a predetermined degree of similarity or higher. No match was defined as not meeting the criteria for an exact match or partial match. As shown in Figure 21, the exact match rate for the comparative example was 80.27%. In contrast, the exact match rate for the example was 85.20%. Furthermore, the partial match rate for the comparative example was 15.82%. In contrast, the partial match rate for the example was 13.98%. In addition, the no match rate for the comparative example was 3.91%. In contrast, the no match rate for the example was 0.82%. That is, by using an input like the example, the rates of partial matches and no matches decreased, and the rate of exact matches increased by the same amount. These results indicate that providing the analysis model with information about the imaging composition can improve the accuracy of the analysis of the imaged object.

1...Image analysis device, 11...Control unit, 12...Storage unit,
13...External interface, 14...Communication interface,
15...Input device, 16...Output device,
81...Program, 91...Storage medium,
20... Instructions for the imaging composition, 30... Captured image,
40...Analysis model, 50...Analysis results, 60...Information
200... Candidate instructions, 250... Parameters,
U: User, C: Imaging device, IT: Image target

Claims (17)

The image is captured after being given instructions for the imaging composition.
The acquired image and the instructions for the imaging composition are given to the analysis model, and the analysis model is made to perform the analysis of the image, thereby obtaining the analysis results of the image from the analysis model.
Outputs information regarding the analysis results of the acquired image.
It includes a control unit configured as follows:
Image analysis device. The aforementioned instruction for the imaging composition is an instruction requesting that the predetermined location of the object to be imaged be positioned according to the markers on the imaging screen.
The image analysis apparatus according to claim 1. The aforementioned instruction for the imaging composition is an instruction requesting that a physical marker be added to a predetermined location on the object to be imaged.
The image analysis apparatus according to claim 1. The instruction for the imaging composition is an instruction to cover a predetermined area of the object to be imaged.
The image analysis apparatus according to claim 1. The instruction for the imaging composition is an instruction requesting that a predetermined part of the object to be imaged be placed at a predetermined position in the image.
The image analysis apparatus according to claim 1. The image analysis apparatus according to claim 1, wherein the instruction for the imaging composition is an instruction to require imaging the object to be imaged in a predetermined orientation. The aforementioned instruction for the imaging composition is an instruction requesting that the object to be imaged be imaged with a predetermined background color.
The image analysis apparatus according to claim 1. The instruction for the aforementioned imaging composition is an instruction to image the target object under predetermined lighting conditions.
The image analysis apparatus according to claim 1. The instruction for the imaging composition is specified when the image is captured.
The image analysis apparatus according to claim 1. Specifying the imaging composition during image acquisition involves selecting one or more instruction candidates from a plurality of instruction candidates and specifying the selected one or more instruction candidates as the imaging composition instruction.
The image analysis apparatus according to claim 9. The instruction for the imaging composition includes one or more parameters,
Specifying the imaging composition when capturing the aforementioned image includes specifying the value of each of the one or more parameters.
The image analysis apparatus according to claim 9. The aforementioned image shows a document as the subject of the image capture.
The image analysis apparatus according to claim 1, wherein the analysis of the image includes identifying at least a portion of the contents of the document. The aforementioned image shows an object as the subject of the image,
The analysis of the aforementioned image includes evaluating at least one of the condition and authenticity of the article.
The image analysis apparatus according to claim 1. The analysis of the aforementioned image includes identifying the object being imaged.
The image analysis apparatus according to claim 1. The aforementioned analysis model is composed of a large-scale generative model.
The image analysis apparatus according to claim 1. A computer-based image analysis method,
The image is captured after being given instructions for the imaging composition.
The acquired image and the instructions for the imaging composition are given to the analysis model, and the analysis model is made to perform the analysis of the image, thereby obtaining the analysis results of the image from the analysis model.
Outputs information regarding the analysis results of the acquired image.
Including,
Image analysis methods. A program that causes a computer to perform an image analysis method,
The aforementioned image analysis method is
The image is captured after being given instructions for the imaging composition.
The acquired image and the instructions for the imaging composition are given to the analysis model, and the analysis model is made to perform the analysis of the image, thereby obtaining the analysis results of the image from the analysis model.
Outputs information regarding the analysis results of the acquired image.
Including,
program.