JP2020024763A - Reflection detection system - Google Patents

Abstract

To generate a reflection detection model with high accuracy capable of detecting a reflection image area indicating a reflection place of light in a captured image even when any captured image is input.SOLUTION: A learning processing method in an AI server includes the steps of: generating a false image B' of an original image A on the basis of the original image A including a reflection area indicating a reflection position of light; evaluating authenticity of the false image B' according to comparison of the false image B' with learning images B1, B2 and B3 generated such that a reflection area g1 in the original image A is distinguishable from other image areas; generating a false image A' of the original image A on the basis of the false image B'; evaluating authenticity of the false image A' according to comparison between the false image A' and the original image A; and generating a learned model used for detecting the reflection area g1 in any captured image on the basis of evaluation results of the authenticity of the false image B' and the false image A'.SELECTED DRAWING: Figure 7

Description

  The present disclosure relates to a reflection detection system.

  Patent Document 1 discloses that in order to specify pixels that can be used for processing such as character reading, a luminance image indicating a luminance value of each pixel included in a captured image is obtained, and a frequency distribution of the luminance value of each pixel is obtained. There is disclosed an image evaluation device that determines a luminance threshold based on the luminance and emphasizes pixels having a high luminance value on the luminance image described above to generate a high luminance partial enhanced image. The image evaluation device specifies a pixel having a luminance exceeding the luminance threshold as a high luminance pixel based on a determination result as to whether a luminance value exceeds a luminance threshold for each of the pixels included in the high luminance partial emphasizing device. .

JP, 2017-162030, A

  However, in Patent Document 1, for example, when a captured image captured by a mobile terminal such as a smartphone includes a portion where light such as illumination light or external light is reflected, light reflection generated in the captured image is included. No consideration is given to detecting an image area.

  The present disclosure is devised in view of the above-described conventional situation, and enables highly accurate reflection detection that can detect a reflection image area indicating a reflection point of light in the captured image even when an arbitrary captured image is input. It is an object of the present invention to provide a reflection detection system capable of generating a model and accurately ensuring the reliability of a reflection image area detected in an arbitrary captured image.

  The present disclosure relates to a reflection detection system in which a server device that holds a captured image to be subjected to a learning process including a reflection image region indicating a light reflection point and a mobile terminal having an imaging unit and a display unit are communicably connected to each other. The server device includes a processor and a memory, wherein the processor generates a first similar image of the captured image based on the captured image in cooperation with the memory, Evaluating the authenticity of the first similar image according to a comparison between the learning image generated such that the reflection image region in the other image region is distinguishable from another image region and the first similar image; Generating a second similar image of the captured image based on the similar image; evaluating the authenticity of the second similar image according to a comparison between the second similar image and the captured image; Each of the similar image and the second similar image Based on the evaluation result of the authenticity of, a reflection detection model used to detect the reflection image region in any captured image is generated, and when an arbitrary captured image captured by the imaging unit is obtained, the reflection detection model Using the to detect the reflected image area in the captured image, generate an output image in which the reflected image area in the captured image is processed to be identifiable from other image areas, and transmit the output image to the mobile terminal. The mobile terminal displays, on the display unit, a result of character recognition of the other image area other than the reflection image area in the output image using the output image transmitted from the server device.

  According to the present disclosure, even when an arbitrary captured image is input, it is possible to generate a highly accurate reflection detection model capable of detecting a reflection image area indicating a reflection point of light in the captured image. The reliability of the detected reflection image area can be ensured accurately.

FIG. 2 is a block diagram showing a hardware configuration of the reflection detection system according to the first embodiment. 4 is a flowchart for explaining an example of an operation procedure of preparation of an original image A and preprocessing. 4 is a flowchart illustrating an example of an operation procedure for generating a learning image B1. 9 is a flowchart illustrating an example of an operation procedure for generating a learning image B2. 9 is a flowchart illustrating an example of an operation procedure for generating a learning image B3. The figure which shows the original image A, the image B0 after pre-processing, and the learning images B1, B2, and B3. Flowchart for explaining an example of an operation procedure of learning of the AI server 5 is a flowchart illustrating an example of an operation procedure of detecting a reflection point of the AI server. Flow chart illustrating an example of a translation operation procedure of a smartphone Diagram showing an example of a shooting screen of a smartphone on which a captured image is displayed Figure showing an example of a smartphone confirmation screen on which a superimposed image is displayed Diagram showing a translation result screen example displayed on a smartphone Diagram showing another translation result screen example displayed on the smartphone A diagram showing an example of a shooting screen of a smartphone on which another captured image is displayed Diagram showing a confirmation screen example of a smartphone on which a superimposed image including a part where a character can be recognized is displayed Figure showing an example of a confirmation screen with a partially recognizable range changed Diagram showing a translation result screen example displayed on a smartphone Diagram showing another translation result screen example displayed on the smartphone

(Process leading to the contents of Embodiment 1)
For example, a traveler such as a foreigner may use a mobile terminal such as a smartphone owned by the traveler at a travel destination to image a subject including a character portion that the traveler wants to check the content of. The portable terminal recognizes a character portion included in the captured image by an operation of a foreigner or the like, and converts the character recognition result into its own native language by a translation application installed in advance. Thereby, a traveler such as a foreigner can check the contents of the character portion included in any captured image captured by the mobile terminal.

  However, as described above, if the light reflection image area exists in the captured image, the character portion cannot be recognized. Therefore, when a character recognizable region (that is, a light reflection image region) is detected in the translation result of a character portion corresponding to an arbitrary captured image displayed on the mobile terminal, the region is clearly specified in the captured image. Then, it would be possible for foreign tourists to provide various applications such as kind translation.

  Therefore, in the following first embodiment, even when an arbitrary captured image is input, it is possible to generate a highly accurate reflection detection model capable of detecting a reflection image area indicating a reflection point of light in the captured image. A learning processing method, a server device, and an example of a reflection detection system that accurately ensure the reliability of a reflection image region detected in a captured image of the present invention will be described.

  Hereinafter, an embodiment that specifically discloses a learning processing method, a server device, and a reflection detection system according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, an unnecessary detailed description may be omitted. For example, a detailed description of a well-known item or a redundant description of substantially the same configuration may be omitted. This is to prevent the following description from being unnecessarily redundant and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the claimed subject matter.

  FIG. 1 is a block diagram illustrating a hardware configuration of the reflection detection system 5 according to the first embodiment. The reflection detection system 5 is configured to include an AI (artificial intelligence) server 10, a smartphone 30, and a translation server 50. The AI (artificial intelligence) server 10, the smartphone 30, and the translation server 50 are communicably connected to each other via a network 70.

  The AI server 10 as an example of the server device has a configuration including a processor 11, an AI processing unit 13, a memory 15, a storage 17, and a communication unit 18.

  The processor 11 is configured using, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA). The processor 11 functions as a controller that controls the operation of the AI server 10, performs control processing for totally controlling the operation of each unit of the AI server 10, data input / output processing with each unit of the AI server 10, (Calculation) processing and data storage processing. The processor 11 operates according to programs and data stored in the memory 15. The processor 11 may use the memory 15 during operation, and temporarily store data or information generated or obtained by the processor 11 in the memory 15.

  The AI processing unit 13 is suitable for, for example, real-time image processing (for example, detection of a light reflection point in a captured image described later and generation of an output image using a learned model) on an arbitrary captured image transmitted from the smartphone 30. It is a processor configured using a GPU (Graphics Processing Unit). The AI processing unit 13 generates a learned model by executing machine learning using the Cycle GAN technique using an original image and a learning image described later, and stores the data of the learned model in the storage 17 (that is, the learned model). Data). The AI processing unit 13 has a memory 13z, for example, reads out the learned model data stored in the storage 17 at the time of executing a process of detecting a light reflection point in an arbitrary captured image transmitted from the smartphone 30, and The model is temporarily expanded and stored in the memory 13z. The AI processing unit 13 receives an arbitrary captured image captured by the smartphone 30 and performs a partial function of the learned model (for example, a false image generator that generates a false image similar to the original image from the original image). A visualized image including an image area of a detected light reflection portion is output using a function, a function of a fake image discriminator that evaluates the authenticity of the generated fake image (for details, see below).

  The memory 15 is configured using, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory), and programs and data necessary for executing the operation of the AI server 10, and further, data generated during the operation. Hold information temporarily. The RAM is a work memory used when the AI server 10 operates, for example. The ROM previously stores and holds, for example, a program and data for controlling the AI server 10.

  The storage 17 is a recording device configured using, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage 17 stores, for example, data or information generated or obtained by the processor 11 or the AI processing unit 13. The storage 17 stores the learned model data generated by the AI processing unit 13 (see FIG. 1).

  The communication unit 18 is connected to the network 70 using, for example, a wired LAN (Local Area Network) or a wireless LAN. The communication unit 18 can communicate with the translation server 50 connected to the network 70, and can communicate with the smartphone 30 carried by a traveler (an example of a user) such as a foreigner. is there. The communication unit 18 receives an arbitrary captured image transmitted from the smartphone 30 (that is, the captured image of an arbitrary subject having a character portion that the traveler wants to check the content of). The communication unit 18 transmits an output image (see below) generated as a result of the light reflection point detection process to the smartphone 30 translation server 50.

  The smartphone 30 is configured to include a processor 31, an imaging unit 32, a display unit 33, an input unit 34, a communication unit 35, and a memory 36. The smartphone 30 is carried, for example, by a traveler such as a foreigner, and is gripped at the time of use. In the smartphone 30, for example, an application (character recognition application) that can execute character recognition processing and an application (translation application) that can execute translation processing are installed in advance so as to be at least executable.

  The processor 31 is configured using, for example, a CPU, an MPU, a DSP, or an FPGA. The processor 31 functions as a controller that controls the operation of the smartphone 30, performs control processing for totally controlling the operation of each unit of the smartphone 30, input / output processing of data with each unit of the smartphone 30, and calculation of data ( Calculation) processing and data storage processing. The processor 31 operates according to programs and data stored in the memory 36. The processor 31 may use the memory 36 during operation, and temporarily store data or information generated or obtained by the processor 31 in the memory 36.

  The imaging unit 32 is configured to include a condensing lens and a solid-state imaging device such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. While the power of the smartphone 30 is on, the imaging unit 32 constantly outputs to the processor 31 the data of the captured video of the subject obtained based on the imaging by the solid-state imaging device. The subject is, for example, an information transmission medium such as a signboard or an advertisement including a character portion that a foreigner or other traveler wants to check the content of, but it is needless to say that the subject is not limited to this information transmission medium.

  The display unit 33 is configured by using, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence), and, in addition to notifying the current state of the smartphone 30, various screens (for example, shooting at the time of imaging by the imaging unit 32). A screen (so-called preview screen), a confirmation screen described later, a screen showing a translation result, and the like are displayed.

  The input unit 34 accepts various input operations by a user (for example, the above-mentioned foreign traveler) and outputs a signal corresponding to the input operation to the processor 31. The display unit 33 and the input unit 34 may be configured by a known touch panel TP.

  The communication unit 35 is configured using a communication circuit that can wirelessly communicate with the AI server 10 and the translation server 50 connected to the network 70. The communication unit 35 is connected to the network 70 via a mobile communication network (not shown) (for example, 4G (fourth generation mobile communication system) and 5G (fifth generation mobile communication system)). The communication unit 35 can communicate with the AI server 10 and the translation server 50 connected to the network 70. The communication unit 35 transmits data of an arbitrary captured image captured by the imaging unit 32 to the AI server 10.

  The memory 36 is configured using, for example, a RAM and a ROM, and temporarily stores programs and data necessary for executing the operation of the smartphone 30, and data or information generated during the operation. The RAM is a work memory used when the smartphone 30 operates, for example. The ROM previously stores and holds, for example, a program and data for controlling the smartphone 30.

  Note that the smartphone 30 is an example of a device having an imaging function and a communication function, and is not limited to a smartphone, and may be a camera, a tablet terminal, a notebook PC, a monitoring camera, or the like that can be connected to the network 70.

  The translation server 50 is configured to include a processor 51, a memory 52, a storage 53, and a communication unit 54. The translation server 50 may be, for example, a cloud server connected to the network 70, or may be configured as an on-premises server installed in an office (not shown) of an operator in which the AI server 10 is arranged, for example. The translation server 50 translates character information corresponding to a character portion in a captured image or an output image transmitted from the smartphone 30 or the AI server 10 into a predetermined language (for example, a language preset by a user of the smartphone 30). Then, character information corresponding to the translation processing result is returned to the smartphone 30.

  The processor 51 is configured using, for example, a CPU, an MPU, a DSP, or an FPGA. The processor 51 functions as a controller that controls the operation of the translation server 50, performs control processing for totally controlling the operation of each unit of the translation server 50, input / output processing of data with each unit of the translation server 50, (Calculation) processing and data storage processing. The processor 51 operates according to programs and data stored in the memory 52. The processor 51 may use the memory 52 during operation, and temporarily store data or information generated or obtained by the processor 51 in the memory 52.

  The memory 52 is configured using, for example, a RAM and a ROM, and temporarily stores programs and data necessary for executing the operation of the translation server 50, and data or information generated during the operation. The RAM is a work memory used when the translation server 50 operates, for example. The ROM previously stores and holds, for example, a program and data for controlling the translation server 50.

  The storage 53 is a recording device configured using, for example, an HDD or an SSD. The storage 53 stores, for example, data or information generated or obtained by the processor 51. Further, the storage 53 includes a dictionary DB 53z in which dictionary data corresponding to a language which is an official language of each country, which is referred to at the time of the translation process, is registered in advance. The translation server 50 periodically communicates with a dedicated dictionary data management server (not shown) connected to the network 70 or another network (not shown), so that the contents of the dictionary DB 53z are stored. May be updated periodically.

  The communication unit 54 is connected to the network 70 using a wired LAN, a wireless LAN, or the like. The communication unit 54 can communicate with the AI server 10 and the smartphone 30 connected to the network 70. When the communication unit 54 receives the character information of the character recognition processing result from the smartphone 30, the received character information is set in advance to correspond to the official language of the user of the smartphone 30 or a predetermined setting set each time. The content is translated into a language, and the character information of the translation is returned to the smartphone 30.

  In the first embodiment, the case where the translation server 50 translates the character information of the character recognition result has been described. However, the smartphone 30 activates the installed translation application and converts the character information of the character recognition result to a predetermined character information. May be translated into language.

  Next, the operation of the reflection detection system 5 according to the first embodiment will be described with reference to the drawings.

  The reflection detection system 5 according to the first embodiment performs character recognition processing on character information included in an image captured by the smartphone 30 of an advertisement or the like in which a character portion is posted, and determines the character information on which the character recognition processing has been performed. Translate to When there is reflection of light such as illumination light or external light in a captured image captured by the smartphone 30, the reflection detection system 5 includes a region including a portion where the light is reflected (hereinafter, referred to as a “reflection region”). In some cases), and outputs a visible light image (hereinafter, may be referred to as an “output image”) including a reflection region that can be identified more than a region other than the reflection region. In the first embodiment, the AI server 10 performs machine learning using a Cycle GAN (cycle gun), which has recently attracted attention as an AI model, and obtains a reflection area ( An AI model (that is, a trained model) for detecting the above (see the above) is generated. In machine learning by Cycle GAN, both a captured image that is an original image and a learning image generated based on the original image are used.

(Generation of learning images)
First, generation of a learning image by the AI server 10 will be described. FIG. 2 is a flowchart illustrating an example of an operation procedure of preparation of the original image A and preprocessing. A user (for example, a foreign traveler or the like; the same applies hereinafter) captures a printed matter (an example of a subject) such as an advertisement using the smartphone 30, and an original image (the original image A in FIG. 6) which is a captured image. Is prepared (S1). For the description of the first embodiment, it is assumed that original image A includes a light reflection region due to external light, illumination light, or the like.

  The user performs predetermined pre-processing on the original image A, and obtains an image B0 after the pre-processing (S2). The predetermined pre-processing for the original image A is performed by, for example, a smartphone 30 or a PC (an image editing application (see below) installed in the illustration 9) (see below). Is painted in a predetermined color.For example, for a captured image displayed on the screen of the smartphone 30, the user uses an image editing application (for example, a drawing tool or image processing software) to display the reflection area. Is painted in red. The pre-processed image (that is, the pre-processed image B0 in FIG. 6) is drawn with the marker region mk filled in red. Is transmitted to the AI server 10. The AI server 10 stores the data of the pre-processed image B0 received from the smartphone 30 in the storage. And stores it in the di-17.

  The AI server 10 generates a plurality of learning images using the preprocessed image B0. Here, an example in which the AI server 10 generates three learning images B1, B2, and B3 will be described, but an arbitrary number of learning images may be generated. By preparing a large number of learning images, the accuracy (that is, learning accuracy) of the process of generating the learned model in the AI server 10 (in other words, updating the learning parameters used for the learned model) is improved.

  FIG. 3 is a flowchart illustrating an example of an operation procedure for generating the learning image B1. The process illustrated in FIG. 3 is executed by, for example, the AI processing unit 13 of the AI server 10. The AI processing unit 13 of the AI server 10 acquires a pixel value of one pixel from the pre-processed image B0 stored in the storage 17 (S11). When acquiring the pixel value of one pixel, the AI processing unit 13 sets two-dimensional coordinates (that is, XY coordinates) for the pre-processed image B0 having the same size as the original image A, for example, and sets the X direction. Then, the pixel value of the corresponding pixel is acquired while moving the acquisition target pixel in the Y direction in pixel units.

  The AI processing unit 13 determines whether or not the pixel is a filled pixel based on the obtained pixel value of one pixel (S12). If the pixel is a filled pixel (S12, YES), the AI processing unit 13 sets the pixel value of this pixel to a predetermined color (for example, set so as to be filled with red), and outputs the pixel of the reflection area (that is, (Corresponding pixels in the learning image B1 generated according to FIG. 3) (S13).

  On the other hand, if the acquired one pixel is not a filled pixel (S12, NO), the AI processing unit 13 sets this pixel to white (for example, to set it to be filled with white) and outputs the pixel in the non-reflection area. (See above) (S14).

  After the processing in step S13 or step S14, the AI processing unit 13 determines whether the one pixel acquired in step S11 is the last pixel (that is, whether the pixel reached the last pixel of the preprocessed image B0). Is determined (S15). If the pixel is not the last pixel (S15, NO), the AI processing unit 13 moves the position of the pixel to be acquired by one pixel in the X direction or the Y direction with respect to the preprocessed image B0 (S16). The processing of the AI processing unit 13 returns to step S11, acquires the next one pixel moved in step S16, and repeats the same processing.

  On the other hand, if the pixel is the last pixel (S15, YES), the AI processing unit 13 converts the image obtained by the series of steps S11, S12, S13, S14, S16, and S15 into the learning image B1 (see FIG. 6). And stored in the memory 13z (S17). Thereafter, the AI processing unit 13 ends the generation processing of the learning image B1.

  FIG. 4 is a flowchart illustrating an example of an operation procedure for generating the learning image B2. 4 is executed by, for example, the AI processing unit 13 of the AI server 10. The AI processing unit 13 of the AI server 10 acquires a pixel value of one pixel from the original image A (S21). The AI processing unit 13 acquires a pixel value of one pixel corresponding to one pixel of the original image A (that is, the XY coordinates are the same) from the preprocessed image B0 (S22). The AI processing unit 13 determines whether one pixel of the pre-processed image B0 acquired in step S22 is a filled pixel based on the acquired pixel value of one pixel (that is, the light reflection area). Is determined) (S23).

  If the pixel is a filled pixel (S23, YES), the AI processing unit 13 calculates a luminance value from one pixel value of the original image A (S24). For example, assuming that the red component is r, the green component is g, the blue component is b, and the luminance value y, the AI processing unit 13 can calculate the luminance value y by the formula of “y = 0.299r + 0.587g + 0.114b”. Yes, and so on. The AI processing unit 13 applies the luminance calculated in step S24 to the R pixel of the output pixel corresponding to one pixel of the original image A (that is, the corresponding pixel in the learning image B2 generated by the operation illustrated in FIG. 4). A value is set (S25). The AI processing unit 13 sets a luminance value of 0 to each of the G and B pixels of the output pixel (S26).

  On the other hand, if one pixel of the pre-processed image B0 acquired in step S22 is not a solid pixel (S23, NO), the AI processing unit 13 outputs the pixel value of one pixel of the original image A to an output pixel (described above). (See S27).

  After the processing in step S26 or step S27, the AI processing unit 13 determines whether the pixel acquired in step S21 is the last pixel (that is, whether it has reached the last pixel of the original image A0) ( S28). If the pixel is not the last pixel (S28, NO), the AI processing unit 13 moves the position of the pixel to be acquired by one pixel in the X direction or the Y direction with respect to the original image A (S29). The process of the AI processing unit 13 returns to step S21, acquires the next one pixel moved in step S29, and repeats the same process.

  On the other hand, if the pixel is the last pixel (S28, YES), the AI processing unit 13 converts the image obtained by the series of processes of steps S21, S22, S23, S24, S25, S26, S27, and S28 into the learning image B2 ( (See FIG. 6) and stored in the memory 13z (S30). Thereafter, the AI processing unit 13 ends the process of generating the learning image B2.

  FIG. 5 is a flowchart illustrating an example of an operation procedure for generating the learning image B3. The process illustrated in FIG. 5 is executed by, for example, the AI processing unit 13 of the AI server 10. The AI processing unit 13 of the AI server 10 acquires a pixel value of one pixel from the original image A (S31). The AI processing unit 13 acquires a pixel value of one pixel corresponding to one pixel of the original image A (that is, the XY coordinates are the same) from the preprocessed image B0 (S32). The AI processing unit 13 calculates a luminance value from the pixel value of one pixel of the original image A acquired in step S31 using, for example, the above-described calculation formula (S33).

  The AI processing unit 13 determines whether one pixel of the preprocessed image B0 acquired in step S32 is a filled pixel based on the acquired pixel value of one pixel (that is, the light reflection area). (S34). If the pixel is a solid pixel (S34, YES), the AI processing unit 13 outputs the output pixel corresponding to one pixel of the original image A (that is, the corresponding pixel in the learning image B3 generated by the operation illustrated in FIG. 5). ), The luminance value of the R pixel is set to the luminance value calculated in step S33 (S35). The AI processing unit 13 sets a luminance value of 0 to each of the G and B pixels of the output pixel (see above) (S36).

  On the other hand, if one pixel of the pre-processed image B0 acquired in step S32 is not a filled pixel (S34, NO), the AI processing unit 13 outputs the R, G, and B pixels of the output pixel (see above). The brightness value calculated in step S33 is set for each (S37).

  After the processing in step S36 or step S37, the AI processing unit 13 determines whether the pixel acquired in step S31 is the last pixel (that is, has reached the last pixel of the original image A) ( S38). If the pixel is not the last pixel (S38, NO), the AI processing unit 13 moves the position of the acquisition target pixel by one pixel in the X direction or the Y direction with respect to the original image A (S39). The process of the AI processing unit 13 returns to step S31, acquires the next one pixel moved in step S39, and repeats the same process.

  On the other hand, if the pixel is the last pixel (S39, YES), the AI processing unit 13 converts the image after the series of processing in steps S31, S32, S33, S34, S35, S36, S37, and S38 into the learning image B3 (FIG. 6). (See S40) and stored in the memory 13z (S40). Thereafter, the AI processing unit 13 ends the process of generating the learning image B3.

  FIG. 6 is a diagram showing an original image A, an image B0 after preprocessing, and learning images B1, B2, and B3. The original image A is a captured image captured by the smartphone 30 based on a user operation with an advertisement, a menu of a restaurant or the like as a subject. In the original image A, there is a reflection area g1 due to light such as illumination light or external light, and character recognition is not possible near the reflection area g1 (in other words, character information cannot be read).

  The image B0 after the preprocessing is an image obtained by performing the preprocessing (see FIG. 2) on the original image A. The image B0 after the preprocessing includes a marker area mk in which the reflection area is filled with red by the user using a drawing tool or image processing software.

  The learning image B1 is an image in which the marker area mk is set to a predetermined color (here, red) and the other areas are set to a background color (white) with respect to the preprocessed image B0. Note that the predetermined color set in the marker area mk may be an arbitrary color such as blue instead of red. Further, the background color is not limited to white, but may be a color that is not much included in the captured image, such as green or blue.

  The learning image B2 calculates a luminance value from the original image A, replaces the R component among the R, G, and B components in the marker area mk with the calculated luminance value, and sets the G and B components to the luminance value 0. , And other areas are pixel values of the original image A.

  The learning image B3 is an image in which, after calculating the luminance value from the original image A, the R component is replaced with the luminance value in the marker area mk, and the R, G, B components are replaced with the luminance value in the other areas.

(Machine learning to generate a trained model)
FIG. 7 is a flowchart illustrating an example of a learning operation procedure of the AI server 10. The process illustrated in FIG. 7 is executed by, for example, the AI processing unit 13 of the AI server 10. The AI processing unit 13 of the AI server 10 sets parameters (hereinafter, referred to as “learning parameters”) used in the AI model (for example, the above-described Cycle GAN) (S51).

  The learning parameter is, for example, a learning rate (that is, a learning rate) when learning a neural network that forms the AI model. In the machine learning according to the first embodiment, for example, learning parameters of an AI model using Cycle GAN are optimized. The AI model using the Cycle GAN includes, for example, a B ′ generator, a false B evaluator, a BB ′ similarity evaluator, an A ′ generator, a false A evaluator, and an AA ′ similarity evaluator. . In the AI model using the Cycle GAN, the original image A, the fake image A 'of the original image A, the learning image B, and the fake image B' of the learning image B are used. In the AI model, the learning parameters of the B ′ generator are optimized. The B 'generator generates a fake image B' from the original image A or the fake image A '. In this learning model, the learning parameters of the A ′ generator are optimized. The A 'generator generates a fake image A' from the learning image B or the fake image B '. As the learning image B, the learning images B1, B2, and B3 shown in FIG.

  The AI processing unit 13 generates a fake image B ′ from the original image A (S52). That is, the AI processing unit 13 inputs the original image A to the AI model B 'generator (fake image generator) to generate a fake image B'. Then, the AI processing unit 13 evaluates the generation accuracy of the fake image B '(S53). Based on the result of this evaluation, the generation accuracy index KB1 serving as the accuracy index of the B ′ generator is updated. The AI processing unit 13 evaluates whether the false image B 'generated by the B' generator is true or false by using the false B evaluator (false image discriminator) (S54). That is, the false B evaluator determines the authenticity of the false image B 'generated by the B' generator. As a result of this determination, the discrimination accuracy index KB2 serving as the accuracy index of the false B evaluator is updated.

  The AI processing unit 13 generates a fake image A 'from the fake image B' (S55). That is, the AI processing unit 13 generates a fake image A 'by inputting the fake image B' to the A 'generator of the AI model. The AI processing unit 13 evaluates the similarity of the generated fake image A ′ (S56). That is, the AA ′ similarity evaluator calculates the similarity between the fake image A ′ and the original image A. As a result of the calculation of the similarity, the reconstruction accuracy index KA3 of the original image A and the reconstructed fake image A 'is updated.

The AI processing unit 13 generates a fake image A ′ from the learning image B (S57). That is,
The AI processing unit 13 inputs the learning image B to the A ′ generator (fake image generator) and generates a fake image A ′. Then, the AI processing unit 13 evaluates the generation accuracy of the fake image A '(S58). Based on the result of this evaluation, the generation accuracy index KA1 serving as the accuracy index of the A ′ generator is updated. The AI processing unit 13 evaluates the falseness of the false image A 'generated by the A' generator using the false A evaluator (false image discriminator) (S59). That is, the fake A evaluator determines whether the fake image A 'generated by the A' generator is true or false. As a result of this determination, the discrimination accuracy index KA2 serving as the accuracy index of the false B evaluator is updated.

  The AI processing unit 13 generates a fake image B 'from the fake image A' (S60). That is, the AI processing unit 13 inputs the fake image A 'to the B' generator and generates the fake image B '. The AI processing unit 13 evaluates the similarity of the generated fake image B ′ (S61). That is, the BB ′ similarity evaluator calculates the similarity between the false image B ′ and the learning image B. As a result of the calculation of the similarity, the reconstruction accuracy index KB3 of the original image B and the reconstructed fake image B ′ is updated.

  The AI processing unit 13 determines a learning parameter of the AI model based on the above-described generation accuracy index KA1, the determination accuracy index KA2, the reconstruction accuracy index KA3, the generation accuracy index KB1, the determination accuracy index KB2, and the reconstruction accuracy index KB3. For example, the learning parameters of the B ′ generator and the learning parameters of the A ′ generator are updated (S62).

  The AI processing unit 13 determines whether or not the learning processing in steps S52 to S62 has been performed using all the original images A and the learning images B (for example, the learning images B1, B2, and B3) (S63). That is, the AI processing unit 13 determines whether or not the data of all the original images A and the learning images B has been learned. Note that the original image A and the learning images B (B1, B2, B3) shown in FIG. 6 are merely examples, and it is desirable to use many original images A and the learning images B in order to improve learning accuracy.

  When there is data that has not been learned (S63, NO), the AI processing unit 13 acquires the next data (S64). The processing of the AI processing unit 13 returns to step S52, and performs the same processing (that is, a series of processing of steps S52, S53, S54, S55, S56, S57, S58, S59, S60, S61, S62, S63, and S64). repeat.

  On the other hand, when all the data has been learned (S63, YES), the AI processing unit 13 generates a learned model (that is, an AI model using the learned Cycle GAN), and generates the learned model. The data is stored in the storage 17 (S65). Thereafter, the AI processing unit 13 ends the learning process illustrated in FIG.

  FIG. 8 is a flowchart illustrating an example of an operation procedure of detecting a reflection point of the AI server 10. The process illustrated in FIG. 8 is executed by, for example, the AI processing unit 13 of the AI server 10. The AI processing unit 13 of the AI server 10 acquires a captured image captured by the smartphone 30 as a detection target image and stores it in the memory 13z (S71). The AI processing unit 13 reads out the learned model data stored in the storage 17, expands it into the memory 13z as an AI network, and takes it in (S72).

  The AI processing unit 13 inputs a detection target image (captured image) to the B ′ generator, which is a part of the learned model, and outputs an image in which the reflection area is visualized (S73). The image in which the reflection area is visualized is an image in which the reflection area is drawn in red and the other areas are drawn in gray, for example, when the AI processing unit 13 executes a process using the learned model (AI model).

  The AI processing unit 13 determines whether it is a non-reflection area or a reflection area based on the intensity ratio of the color components of the image, and acquires reflection area information (S74). The image of the non-reflection area is subjected to each of the character recognition processing and the translation processing as described later. The image of the reflection area is excluded from the character recognition processing and the translation processing. Thereafter, the AI processing unit 13 ends the AI reflection detection processing shown in FIG.

(Translation operation of smartphone)
FIG. 9 is a flowchart illustrating an example of a translation operation procedure of smartphone 30. The process illustrated in FIG. 9 is mainly executed by, for example, the processor 31 of the smartphone 30. The processor 31 of the smartphone 30 activates the character recognition / translation application upon receiving a user operation (S81). When the user performs a shutter operation (that is, an imaging start operation) on a subject such as an advertisement, the imaging unit 32 captures an image of the subject. The processor 31 acquires the captured image GZ1 (see FIG. 10) captured by the imaging unit 32, and stores it in the memory 36 (S82). The communication unit 35 transmits the captured image GZ1 stored in the memory 36 to the AI server 10 via the network 70 (S83).

  FIG. 10 is a diagram illustrating an example of the shooting screen GM1 of the smartphone 30 on which the captured image GZ1 is displayed. It is assumed that, in the captured image GZ1, for example, reflection areas g1 due to illumination light appear at two places. Further, on the photographing screen GM1, a rectangular window wk1 is displayed so as to be superimposed on the captured image GZ1. The captured image GM1 is hidden behind the rectangular window wk1 and is not displayed, but the captured image GZ1 includes character information of coffee and tea (see FIG. 12B). Further, a shutter icon st indicating a shutter button of the camera (that is, an imaging start button) is displayed on the shooting screen GM1.

  The communication unit 18 of the AI server 10 receives a captured image from the smartphone 30. The AI processing unit 13 performs the AI reflection detection process illustrated in FIG. 8 on the received captured image to acquire reflection area information. The communication unit 18 transmits the reflection area information obtained by the AI processing unit 13 to the smartphone 30.

  The communication unit 35 of the smartphone 30 receives the reflection area information from the AI server 10 via the network 70 (S84). The processor 31 superimposes the reflection position mc represented by a specific color (for example, red) on the captured image stored in the memory 36 based on the received reflection region information, and superimposes the reflection position mc. The image GZ2 is generated and displayed on the display unit 33 (S85).

  FIG. 11 is a diagram illustrating an example of the confirmation screen GM2 of the smartphone 30 on which the superimposed image GZ2 is displayed. The processor 31 performs character recognition on the superimposed image GZ2 on which the reflection positions are superimposed (S86). The processor 31 stores the result of the character recognition process in the memory 36. Recognized characters are provided with a character hm as a marking indicating that the characters have been recognized. When the character hanging hm is performed, the display form of the character displayed on the screen of the display unit 33 changes. For example, the color of the character changes from black before character recognition to gray surrounding the character.

  Further, the processor 31 displays a rectangular window wk2 below the confirmation screen GM2, and displays a message for confirming the presence or absence of translation in the rectangular window wk2. Here, a message “Translate the display. Is it OK?” Is displayed in the display area set below the screen of the touch panel TP. Further, below the screen of the touch panel TP, a YES button 34z and a NO button 34y are arranged as the input unit 34. When performing translation as a result of character recognition, the user presses the YES button 34z. In addition, when not performing the translation, the user presses the NO button 34y.

  The processor 31 accepts the operation of the user and determines whether or not to start the translation (S87). When starting the translation, the communication unit 35 transmits the character information obtained as a result of the character recognition to the memory 36 to the translation server 50 connected to the network 70 according to the instruction of the processor. The communication unit 54 of the translation server 50 receives the character information transmitted from the smartphone 30. The processor 51 of the translation server 50 refers to the dictionary DB 53z of the storage 53, and translates the character information in a language of a country designated in advance (for example, the foreign language of the foreigner himself). The communication unit 54 transmits the result of the translation process to the smartphone 30.

  The communication unit 35 of the smartphone 30 receives a translation result from the translation server 50. The processor 31 displays the translation result on the screen of the display unit 33 (S88). Here, the translation server performed the translation, but the smartphone 30 may activate the installed translation application and perform the translation on its own device.

  FIG. 12A is a diagram illustrating an example of the translation result screen GM3 displayed on the smartphone 30. The translation result is displayed in a region arranged below the translation result screen GM3 and surrounded by the rectangular window wk3. Here, the translation results "Curry" and "Ooolong" are displayed for the character information "Curry" and "Ourongcha", respectively. In addition, no translation is performed on the images of “Takoyaki” and “Yakisoba” on which the reflection position mc is superimposed and the characters are not recognized, so that nothing is displayed. Here, the language is translated from Japanese to English, but the language before translation and the language after translation can be any combination such as Japanese, English, Chinese, German, French and the like. The translation application determines the nationality of the owner set on the smartphone 30 and performs translation in the language of the corresponding country.

  The user can save a translation result by performing a predetermined operation on the touch panel TP. As the predetermined operation, for example, a double tap operation is performed on an area surrounded by a rectangular window wk3 displayed on the translation result screen GM3.

  The processor 31 accepts the operation of the user and determines whether to save the translation result (S89). When storing the translation result, the processor 31 stores the translation result in the memory 36 (S90). The processor 31 determines whether an application end operation has been performed (S91). If the application end operation has not been performed, the process returns to step S82. On the other hand, when the application ending operation is performed, or when the translation result is not stored in step S89, the processor 31 ends the process as it is.

(Other translation result screen)
FIG. 12B is a diagram illustrating an example of another translation result screen GM4 displayed on the smartphone 30. No rectangular window is displayed on the translation result screen GM4, and the character recognition result image GZ4 and the translation result image GZ5 are displayed in comparison. The character recognition result image GZ4 includes “curry”, “oolong tea”, “coffee”, and “tea”, which are character information on which character recognition has been performed. The translation result image GZ5 includes translated character information “Curry”, “Ooolong”, “Coffee”, and “Black tea”.

(Example of other screen display on smartphone)
As another usage example, a case where a user captures an image of a meal menu using the smartphone 30 will be described. FIG. 13 is a diagram illustrating an example of a shooting screen GM6 of the smartphone 30 on which another captured image GZ6 is displayed. Similarly to the shooting screen GM1 shown in FIG. 10, the shooting image GM6, the rectangular window wk6, and the shutter icon st are displayed on the shooting screen GM6. The captured image GZ6 includes character information such as a meal menu, chicken curry, pork curry, beef curry, and a dring menu. In the image near the chicken curry, a light reflection area g2 overlaps with the “ray” portion of the chicken curry.

  FIG. 14A is a diagram illustrating an example of a confirmation screen GM7 of the smartphone 30 on which a superimposed image GZ7 including a range in which some characters can be recognized is displayed. As a result of performing the character recognition on the captured image GZ6, the meal menu, the pork curry, the beef curry, and the dring menu were recognized on the confirmation screen GM7. Recognized characters are provided with a character hm as a marking indicating that the characters have been recognized. As described above, when the character hm is applied, the display form of the character displayed on the screen of the display unit 33 changes.

  On the other hand, the reflection position mc is superimposed and displayed on the area including the chicken curry. In this area, the reflection position mc is superimposed and displayed in the vicinity. Also, a part of the chicken curry, which is not the entire chicken curry but part of which can be recognized, is recognizable by the marker mr. Here, the range in which some characters can be recognized is the portion of “chicken curry” in chicken curry. The range of “Chickenka” is, for example, shaded in orange (indicated by hatching in the figure) as a marker mr. Further, left and right cursors ks are displayed on the touch panel TP so as to sandwich the portion of “chickenka”. When the user drags the cursor ks with a finger, for example, the range in which some characters can be recognized is changed.

  FIG. 14B is a diagram illustrating an example of the confirmation screen GM8 in which the range in which some characters can be recognized has been changed. The user determines that the translation is “misplaced” even if “Chickenka” is translated, and moves the cursor ks one character to the left in the figure with the finger. The range in which some characters can be recognized changes to the part of “chicken”. Thereby, when the chicken is translated, the chicken curry is associated.

  14A and 14B, similarly to FIG. 11, rectangular windows wk7 and wk8 are displayed below confirmation screens GM7 and GM8, respectively, and a message for confirming the presence or absence of translation is displayed. When the user presses the YES button 34z displayed below the touch panel TP, translation is performed on the confirmation screen GM8.

  FIG. 15A is a diagram illustrating an example of the translation result screen GM9 displayed on the smartphone 30. The translation result is displayed below the translation result screen GM9 in a region surrounded by the rectangular window wk9. Here, the translation results of “food menu”, “chiken”, “poke curry”, “beef curry”, “beef curry” and “food menu”, “chicken”, “pork curry”, “beef curry”, and “dling menu”, respectively, are described. "link menu" is displayed.

(Other translation result screen)
FIG. 15B is a diagram illustrating an example of another translation result screen GM10 displayed on the smartphone 30. The area surrounded by the rectangular window wk10 displayed below the translation result screen GM10 is blank. The translation result screen GM10 overwrites the food information, such as the meal menu, chicken curry, pork curry, beef curry, and dring menu, and the translation results “food menu”, “chiken”, “poke curry”, “Beef curry” and “drink menu” are displayed. However, the area near the reflection position mc is not translated and is displayed as it is.

  As described above, even when the reflection position is included in the image captured by the smartphone 30, the translation result is displayed so that the user can read it.

  As described above, in the learning processing method in the AI server 10 according to the first embodiment, the original image A (the learning processing target) including the reflection area g1 (an example of the reflection image area) indicating the light reflection position (an example of the reflection point) is shown. A step of generating a fake image B ′ of the original image A (an example of a first similar image) based on the captured image). The learning processing method is based on a comparison between learning images B1, B2, and B3 generated so that the reflection area g1 in the original image A (an example of a captured image) can be distinguished from other image areas, and a false image B ′. And evaluating the authenticity of the false image B ′. Further, the learning processing method includes a step of generating a fake image A ′ (an example of a second similar image) of the original image A based on the fake image B ′. Further, the learning processing method includes a step of evaluating the authenticity of the fake image A ′ according to the comparison between the fake image A ′ and the original image A. The learning processing method is based on a learned model (an example of a reflection detection model) used for detecting a reflection area g1 in an arbitrary captured image based on the evaluation results of the fake images B ′ and A ′. ).

  Thus, even when an arbitrary captured image is input from the smartphone 30, the AI server 10 can generate a highly accurate reflection detection model capable of detecting a reflection image area indicating a reflection point of light in the captured image, The reliability of the reflection image area detected in any captured image can be ensured accurately.

  In the learning processing method, the step of generating a learned model includes a learning parameter (an example of a parameter) used by the learned model based on the evaluation result of the authenticity of each of the false image B ′ and the false image A ′. ) And generating a trained model using the updated learning parameters. Thereby, the AI server 10 can generate a highly accurate learned model in which learning parameters are updated based on the evaluation result of the authenticity of the false image and the original image, and can improve the learning effect of the learned model.

  Further, in the learning processing method, the step of generating the fake image B ′ is performed when there are a plurality of original images A (an example of a captured image to be subjected to learning processing), and the fake image B ′ corresponding to each of the original images A is provided. Generating. Thereby, the AI server 10 can generate a plurality of fake images corresponding to a plurality of different original images A, and can update the learning parameters for each of the original images A, thereby further improving the accuracy of the reliability of the learned model. Can be improved.

  In the learning processing method, red (an example of the first color) is added to the reflection area g1 in the original image A (an example of the captured image), and the reflection area g1 is added to the image area other than the reflection area g1 in the original image A. The method further includes a step of generating a learning image B1 by giving white (an example of a second color). Accordingly, the AI server 10 can easily generate a learning image in which the light reflection region and the other region are clearly identified in the captured image input from the smartphone 30.

  In the learning processing method, the pixel value of each of the R pixels (an example of any one of the pixels) forming the reflection area g1 in the original image A (an example of the captured image) corresponds to the pixel value in the original image A The learning image is set by setting the luminance value of the pixel and setting the pixel value of the corresponding pixel in the original image A to the pixel value of each pixel constituting an image area other than the reflection area g1 in the original image A. The method further includes generating B2. Accordingly, the AI server 10 can easily generate a learning image in which the light reflection region and the other region are clearly identified in the captured image input from the smartphone 30.

  In the learning processing method, the pixel value of each of the R pixels (an example of any one of the pixels) constituting the reflection area g1 in the original image A (an example of the captured image) corresponds to the pixel value in the original image A. The luminance value of the pixel is set, and the pixel value of each of the RGB pixels (an example of all colors) constituting an image area other than the reflection area g1 in the original image A corresponds to the pixel value in the original image A. And generating the learning image B3 by setting the pixel values of. Accordingly, the AI server 10 can easily generate a learning image in which the light reflection region and the other region are clearly identified in the captured image input from the smartphone 30.

  In addition, the reflection detection system 5 according to the first embodiment enables the AI server 10 (an example of a server device) and the smartphone 30 (an example of a mobile terminal) having the imaging unit 32 and the display unit 33 to communicate with each other. Connected. When acquiring an arbitrary captured image captured by the imaging unit 32, the AI server 10 uses a learned model (an example of a reflection detection model) to determine a light reflection area (an example of a reflection image area) in the captured image. At the same time as detecting, an output image in which the light reflection area in the captured image is processed so as to be distinguishable from other image areas is generated and transmitted to the smartphone 30. Using the output image transmitted from the AI server 10, the smartphone 30 displays a result of character recognition of an image area other than the light reflection area in the output image on the display unit 33.

  Thereby, the user using the smartphone 30 (for example, a traveler such as a foreigner) transmits a captured image whose subject is an advertisement or the like whose content is to be confirmed to the AI server 10 and processes the captured image by the AI server 10. By causing the smartphone 30 to recognize and translate the result, the translation result of the character information recognized as the character portion can be grasped. In other words, the reflection detection system 5 can provide a user with a kind character recognition and translation application for a foreign traveler or the like, and can improve the convenience of the user appropriately.

  Although various embodiments have been described with reference to the accompanying drawings, the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various changes, modifications, substitutions, additions, deletions, and equivalents can be made within the scope of the claims. It is understood that it belongs to the technical scope of the present disclosure. Further, the components in the above-described various embodiments may be arbitrarily combined without departing from the spirit of the invention.

  INDUSTRIAL APPLICABILITY The present disclosure is useful because it can generate a highly accurate reflection detection model that can detect a reflection image area indicating a light reflection point in a captured image.

5 Reflection detection system 10 AI server 13 AI processing unit 30 Smartphone 32 Imaging unit 33 Display unit

Claims (1)

A reflection detection system in which a server device holding a captured image of a learning processing target including a reflection image region indicating a light reflection point and a mobile terminal having an imaging unit and a display unit are communicably connected to each other,
The server device,
A processor and a memory,
The processor, in cooperation with the memory,
Generating a first similar image of the captured image based on the captured image;
In accordance with a comparison between the learning image and the first similar image generated such that the reflection image area in the captured image is identifiable with another image area, evaluates the authenticity of the first similar image,
Generating a second similar image of the captured image based on the first similar image;
Evaluating the authenticity of the second similar image according to the comparison between the second similar image and the captured image,
Based on the evaluation results of the authenticity of each of the first similar image and the second similar image, generate a reflection detection model used for detecting the reflection image region in an arbitrary captured image,
When an arbitrary captured image captured by the imaging unit is acquired, the reflected image region in the captured image is detected using the reflection detection model, and the reflected image region in the captured image is replaced with another image. Generate an output image processed to be identifiable as the area and send it to the portable terminal,
The mobile terminal,
Using the output image transmitted from the server device, a result of character recognition of the other image region other than the reflection image region in the output image is displayed on the display unit,
Reflection detection system.

Images