JP6815712B1 - Image processing system, image processing method, image processing program, image processing server, and learning model - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract an image of a predetermined object from a moving image without detecting an edge of an imaged object, and to appropriately extract an image of the object regardless of the difference in the background in which the object is arranged. It provides a system, an image processing method, an image processing program, an image processing server, and a learning model. An image processing system 1 has a coordinate acquisition unit 16 that acquires coordinates at a predetermined position of an object based on a moving image of a moving image of an object, and an image area including the object based on the coordinates. The image area extraction unit 18 for extracting from the constituent image is provided, the coordinate acquisition unit 16 acquires the coordinates of a predetermined position of the object based on the image area, and the image area extraction unit 18 obtains the acquired coordinates for the moving image constituent image. Extract the object image area of the object by projecting to. [Selection diagram] Fig. 1

Description

The present invention relates to an image processing system, an image processing method, an image processing program, an image processing server, and a learning model. In particular, the present invention relates to an image processing system, an image processing method, an image processing program, an image processing server, and a learning model capable of appropriately extracting a predetermined object in a moving image.

Conventionally, it is a line segment extraction device that extracts a line segment from an image, and has an edge detection unit that detects an edge from the image, a plurality of first parallel lines extending in a first direction in the image at predetermined intervals, and an edge. The first intersection identification part for finding the intersection of, and the first intersection connection part connecting the intersections on the first parallel lines of each pair of two adjacent first parallel lines with a straight line connection line, and the intersection. There is known a line segment extraction device including a first line segment specifying unit that extracts a set of coupling lines consisting of a plurality of coupling lines connected by a line and having an angle difference in the stretching direction within a predetermined range as a line segment (. For example, see Patent Document 1.). According to the line segment extraction device described in Patent Document 1, the line segments included in the image can be extracted at high speed.

Japanese Unexamined Patent Publication No. 2018-181244

However, the line segment extraction device described in Patent Document 1 is premised on detecting the edge of an object, and when the object has a rectangular shape, it is required to extract at least three sides of the object. Further, the line segment extraction device described in Patent Document 1 extracts all rectangular areas regardless of the type of the object as long as it is a rectangular area. Further, in the line segment extraction device described in Patent Document 1, it is difficult to recognize the edge of the object depending on the combination of the object and the background on which the object is placed (for example, the color of the object and the background color are different). If they are approximately the same, it may be difficult to recognize the edges), in which case it may be difficult to recognize the existence of the object.

Therefore, an object of the present invention is that an image of a predetermined object can be extracted from a moving image without detecting the edge of the captured object, and the image of the object can be appropriately used regardless of the background difference in which the object is arranged. It is an object of the present invention to provide an image processing system, an image processing method, an image processing program, an image processing server, and a learning model that can be extracted.

In order to achieve the above object, the present invention has a coordinate acquisition unit that acquires coordinates of a predetermined position of an object based on a moving image of a moving image of an object, and an image area including the object based on the coordinates. It is provided with an image area extraction unit that extracts from the constituent image, the coordinate acquisition unit acquires the coordinates of a predetermined position of the object based on the image area, and the image area extraction unit projects the acquired coordinates onto the moving image constituent image. An image processing system that extracts the object image area of the object is provided.

Further, in order to achieve the above object, the present invention is an image processing method for an image processing system, which is a coordinate acquisition step of acquiring coordinates of a predetermined position of an object based on a moving image of a moving image of an object. , An image area extraction step of extracting an image area containing an object from a moving image constituent image based on the coordinates, a step of acquiring the coordinates of a predetermined part of the object based on the image area, and a moving image constituent image of the acquired coordinates. An image processing method including a step of projecting an image onto an object and extracting an object image area of the object is provided.

Further, in order to achieve the above object, the present invention is an image processing program for an image processing system, and coordinates for acquiring the coordinates of a predetermined position of an object on a computer based on a moving image of a moving image of an object. The acquisition function, the image area extraction function that extracts the image area containing the object from the video composition image based on the coordinates, the function that acquires the coordinates of a predetermined part of the object based on the image area, and the acquired coordinates. An image processing program is provided that realizes a function of projecting a moving image and extracting an object image area of an object.

Further, in order to achieve the above object, the present invention has a coordinate acquisition unit that acquires coordinates of a predetermined position of an object based on a moving image of a moving image of an object, and an image area that includes the object based on the coordinates. The image area extraction unit is provided with an image area extraction unit that extracts the image from the moving image configuration image, the coordinate acquisition unit acquires the coordinates of a predetermined position of the object based on the image area, and the image area extraction unit converts the acquired coordinates into the moving image composition image. An image processing server is provided that projects and extracts the object image area of the object.

Further, in order to achieve the above object, when a captured image is input, the present invention performs one or more corners of a predetermined object in order to identify whether or not the object included in the captured image is a predetermined object. A learning model that causes a processor to function so as to output one or more rectangular areas at the center. The learning model includes an image containing a predetermined object, a background image in which a predetermined object can be arranged, and a predetermined object. The combination of the included image and the background image is learned as teacher data, and in the training, one or more rectangular areas centered on the corners of a predetermined object, from the center to the outer edge of the image containing the predetermined object. A rectangular region having a size in which a side whose vertical line is a straight line having the shortest length is in contact with the outer edge of the image is formed, and a predetermined rectangular region in the image is used by using the formed rectangular region and the coordinates of the center of the rectangular region. A learning model for identifying objects is provided.

According to the image processing system, the image processing method, the image processing program, the image processing server, and the learning model according to the present invention, the image of a predetermined object can be extracted from the moving image without detecting the edge of the captured object. At the same time, it is possible to provide an image processing system, an image processing method, an image processing program, an image processing server, and a learning model that can appropriately extract an image of the object regardless of the difference in the background in which the object is arranged.

It is a schematic diagram of the image processing system which concerns on this embodiment. It is a functional block diagram of the image processing system which concerns on this embodiment. It is a schematic diagram of the labeling method of a predetermined object in the learning model generated by the learning model generation part which concerns on this embodiment. It is the schematic of the 1st process of the processing of the image processing system which concerns on this embodiment. It is a schematic diagram of the reason why the margin area is provided. It is a schematic diagram of the 2nd process of the processing of the image processing system which concerns on this embodiment. It is a processing flow diagram of the image processing system which concerns on this embodiment.

[Embodiment]
FIG. 1 shows an outline of an image processing system according to an embodiment of the present invention.

[Overview of image processing system 1]

The image processing system 1 according to the present embodiment captures a moving image of a region including a predetermined object, and automatically and / or automatically obtains the predetermined object and / or the information described in the predetermined object from the captured moving image. It is a system that extracts properly. For example, the image processing system 1 has a plurality of receipts (the plurality of receipts may have different shapes / sizes and surface description styles), business cards, other pieces of paper, and square objects in the imaging region. When a moving image is taken of a state in which an area (for example, a button on a keyboard of a smartphone or a personal computer) exists, a specific object from these, for example, the plurality of receipts and / or the plurality of receipts. The information described in can be automatically extracted in real time and converted into digital information that can be used in a computer.

For example, as shown in FIG. 1A, the image processing system 1 includes an information terminal 2 having a camera that captures a moving image in a predetermined imaging region, and a server 3 that executes predetermined information processing. An example will be described in which 2 and the server 3 are connected by a communication network 4 so as to be capable of bidirectional communication. Here, as an example, a case where the predetermined object extracted by the image processing system 1 according to the present embodiment is a receipt (receipt) having various shapes and sizes and created in various formats and formats. explain.

For example, suppose a plurality of objects (for example, object 80 and object 82) are arranged on a desk 90. The plurality of objects may be arranged by the user at a predetermined location, for example. Then, the image processing system 1 captures a moving image of a region including these with the camera of the information terminal 2. In the example of FIG. 1A, the object 80 (for example, a business card) and the object 82 (for example, a receipt) are arranged on the desk 90. A part of the object 82 may be bent. Then, the image processing system 1 extracts one or more moving image constituent images constituting the moving image from the moving image in which a plurality of objects are captured. Subsequently, the image processing system 1 performs resizing processing on each of the extracted one or more moving image constituent images to generate one or more resized images.

Subsequently, the image processing system 1 acquires the coordinates of a predetermined position of the object to be extracted from the resized image. These coordinates are the coordinates of the predetermined location in the resized image. In this case, the image processing system 1 prepares in advance a learning model for determining whether or not the object included in the image is a predetermined object. For example, when the object to be extracted is a receipt, this learning model recognizes the receipt as a receipt and recognizes the receipt if the image contains an object different from the receipt and the receipt. It is a learning model that can be used to recognize that an object different from is not a receipt.

Here, in the present embodiment, a learning model including an association between one or more rectangular areas (that is, a bounding box) centered on one or more predetermined points of the object to be extracted and the category of the object is prepared in advance. prepare. That is, instead of associating the rectangular area surrounding the entire object to be extracted with the category of the object as in the conventional learning model, a plurality of rectangles surrounding a plurality of parts of one object to be extracted. In this embodiment, a learning model including an association between a region and a category of the object is constructed and used. For example, in the learning model, four square areas centered on the four corners of the receipt are set as one set, the one set is associated with the receipt which is the category of the object, and when the video composition image is input, the video configuration is performed. This is a learning model for outputting the image of the area occupied by the receipt of the image and / or the coordinates of the four corners.

As an example, this learning model is generated based on information about a large number of images of a predetermined object acquired in advance, feature quantities such as corners of a predetermined object and feature points, and a background image in which the object can be arranged. This is a learning model for determining whether or not the object included in the moving image is a predetermined object. The image processing system 1 uses a table that stores information about the feature amount of the object in association with an identifier that identifies the predetermined object, and each of the one or more objects included in the moving image is predetermined. You may judge whether it is an object or not. However, in the present embodiment, it is preferable to determine whether or not the object is a predetermined object by using a learning model from the viewpoint of flexibly, quickly, and accurately corresponding to a predetermined object of various shapes and sizes.

Then, the image processing system 1 uses the learning model to acquire the coordinates of a predetermined position of the object to be extracted from the resized image, for example, the coordinates of the four corners when the object has a rectangular shape. In this case, the image processing system 1 uses a learning model to determine whether or not the object is a receipt based on a square area centered on each of the four corners, and / or 4 of the receipt. Get the coordinates of the corner. Further, since the image processing system 1 captures a moving image, for example, when the moving image is captured while the information terminal 2 is moved, the moving image constituent image does not include the entire object to be extracted. Since the constituent image can also be included, a resized image that does not include the entire object can be generated for the resized image as well. Therefore, the image processing system 1 uses a learning model to select a resized image that includes all the coordinates of the predetermined portion of the object to be extracted, and acquires the coordinates of the predetermined portion of the object from the selected resized image. ..

In the following description, for simplification of the description, the case where the image processing system 1 mainly acquires the coordinates of the four corners of the object and executes the processing will be described, but the image processing system 1 is a part of the object. The coordinates of the corners of can be obtained, and the coordinates of the remaining corners can be estimated and used. That is, the image processing system 1 can acquire some coordinates without acquiring all the coordinates of the four corners of the object 82. In this case, the image processing system 1 can acquire the coordinates of a part of the corners of the object 82, and can estimate the corners for which the coordinates have not been acquired from the coordinates of the acquired corners (for example, three corners). When the coordinates are acquired, the coordinates of the remaining one corner can be estimated, the coordinates of the two diagonal corners can be acquired, and the coordinates of the remaining two corners can be estimated.)

Specifically, in the example of FIG. 1B, the image processing system 1 has four corners (that is, corners 150, corners 152, corners 154, and corners) of the object 82 (that is, a receipt) included in the resized image 100. Obtain at least a part of the coordinates of 156). On the other hand, the image processing system 1 does not acquire the coordinates of the four corners of the object 80 (that is, the business card) that is not the extraction target by using the learning model. The image processing system 1 displays the resized image when a part of the receipt is not included in the resized image, that is, when a part of the four corners of the receipt is not included in the resized image. Coordinates may be acquired using a resized image that includes all four corners of the receipt without using it. Further, in the image processing system 1, even if a part of the object 82 is bent (that is, even if a part of the object 82 is lifted from the desk 90), the corner of the object 82 is resized. As long as it is included in the image 100 or contains some corners of the object 82, either get the coordinates of the four corners of the object 82 from the resized image 100, or get the coordinates of some corners and some corners. Get the coordinates of the remaining corners estimated from the coordinates.

In the learning model used by the image processing system 1, a learning model can be constructed by executing learning in which images of objects to be extracted are superimposed on various background images. As a result, in the image processing system 1, even if the outer edge of the object 82 is difficult to recognize due to the color of the desk 90 as the background, the coordinates of the predetermined position of the object 82 can be appropriately acquired.

Subsequently, the image processing system 1 projects the coordinates acquired from the resized image 100 onto the original moving image constituent image (that is, the resized image is generated from the moving image constituent image) that is the generation source of the resized image. (For example, the coordinate 150a, the coordinate 152a, the coordinate 154a, and the coordinate 156a shown in FIG. 1C) are used, and an image region including the object 82 to be extracted from the original moving image configuration image is included. To extract. In this case, the image processing system 1 may extract an image area including a predetermined margin area around the object 82.

Then, the image processing system 1 resizes the extracted image area to generate the resized image area. Next, the image processing system 1 uses the learning model again to acquire the coordinates of a predetermined position of the object to be extracted from the resized image area again. These coordinates are the coordinates of a predetermined position in the resized image area. Subsequently, the image processing system 1 uses the coordinates obtained by projecting the coordinates acquired from the resized image area onto the original moving image constituent image from which the original image area that is the generation source of the resized image area is extracted. The object image area of the object 82 to be extracted is extracted from the original moving image. As a result, the image processing system 1 can appropriately extract the image of the object 82 to be extracted captured in the moving image in real time. Here, the image processing system 1 may extract the object image area after performing predetermined image processing. For example, if a part of the receipt that is an object is bent and the part is raised from the plane on which the receipt is placed, the text or figure displayed in the raised part is distorted in the object image area. Etc. may occur. Therefore, the image processing system 1 applies image processing for removing the distortion or the like to the object image area. Then, the image processing system 1 stores the object image area as data that can be read by, for example, optical character recognition (OCR).

Further, the image processing system 1 can read the information written on the surface of a predetermined object included in the moving image based on the data, and output the read contents to a display unit or the like of the information terminal 2. For example, when a predetermined object is a receipt, the image processing system 1 actually reads the OCR readable data generated by capturing a moving image including the receipt in the imaging region, and reads the reading result of the information terminal 2. It may be output to a display unit or the like. In this case, for example, when the predetermined object is a receipt, the image processing system 1 sets the specific contents of each receipt, such as the date and issuer of the receipt, the amount of money, and the contents to be sold. It can be output. Further, the image processing system 1 can store the read information, and the stored information can be handed over to an accounting system, a household account book system, or the like other than the image processing system 1 (the image processing system 1 can store the read information. It may be delivered directly to an accounting system or the like other than the image processing system 1).

As a result, according to the image processing system 1, it is not necessary to image a plurality of objects one by one or scan them with a scanner, and only image a state in which a plurality of objects are arranged on a desk or the like as a moving image. Therefore, it is possible to identify each of a plurality of objects and appropriately extract information on the surface of each object. Therefore, when the object is, for example, a receipt created in various shapes and styles, an accounting office that requires processing of a huge number of receipts, a sole proprietor who needs to process a large number of receipts, etc. Alternatively, it is possible to reduce the labor of processing accounting and accounting of various people who create household accounts and improve usability.

In particular, the image processing system 1 according to the present embodiment extracts a moving image constituent image (original image) from the captured moving image, resizes the extracted moving image constituent image to generate a resized image, and extracts the object to be extracted from the resized image. The first step of acquiring the coordinates of a predetermined location, projecting the acquired coordinates onto the moving image constituent image (original image) to extract the image area including the object to be extracted, and resizing this image area again. A resized image area is generated, the coordinates of a predetermined part of the object to be extracted are acquired from the resized image area, and the acquired coordinates are projected onto the video constituent image (original image) to extract the object image area of the object to be extracted. The image of the object to be extracted (that is, the object image area) is extracted through the second step. By going through the first step and the second step, even if it is difficult to detect the edge of the object, the image of the object can be extracted with high accuracy.

Here, in the present embodiment, a learning model is constructed by using the bounding boxes of a plurality of parts instead of the entire object to be extracted. As a result of diligent research by the present inventor, this is done by constructing and using a learning model using a plurality of bounding boxes centered on a characteristic part of the object, rather than using a bounding box containing the entire object. This is the result of finding that the object to be extracted can be extracted with extremely high accuracy and the processing speed of the system can be improved.

That is, in the first step, the image processing system 1 uses a plurality of bounding boxes centered on a plurality of feature portions (for example, corners) of a predetermined object in an image area including a predetermined object from the moving image constituent image. In the second step, the object image area containing a predetermined object is precisely extracted again using a plurality of bounding boxes centered on a plurality of feature portions of the object based on the coarsely extracted image area. Extract to. That is, the bounding box area estimation itself for the Ojibject contains an error. Therefore, in the present embodiment, the error is reduced by repeating the process using the bounding box (that is, at least two steps of the first step and the second step are executed), and the object is detected with high accuracy. can do. As an example of detecting an object by the bounding box, a Single Shot MultiBox Detector (SSD) that detects an object in an image with a single deep neural network can be used. As a result, according to the image processing system 1, where there are various rectangular objects (for example, business cards, receipts, keyboard buttons, smartphones, etc.), the objects to be extracted (receipt in the above example). By constructing a learning model for the book) in advance, the object to be extracted can be appropriately extracted from the moving image, and the detection / extraction of an unintended rectangular area can be prevented.

In this embodiment, the objects are objects having the same shape or different shapes from each other and having a planar shape. The shape of the object is not particularly limited, but may be, for example, a four-sided shape, and the four corners or at least a part of the four sides may be missing. Further, the shape of the object is not limited as long as it has a corner (that is, a corner), and may be a polygon such as a triangle, a pentagon, or a hexagon, or even if a part of the object includes an arc shape. Good. Furthermore, the size of the object is not particularly limited. Then, various types of information (text information, graphic information, handwritten characters, numbers, figures, etc.) may be printed, printed, and / or described on the surface of the object in various formats. Examples of objects include, but are not limited to, quotations, invoices, receipts, and / or business cards. When the object is a receipt, etc., the information written on the surface of the object includes information such as the issue date, issue time, address, amount, description, issuer name, and / or the issuer's telephone number. Can be mentioned. Therefore, the moving image captured by the image processing system 1 according to the present embodiment includes images of a plurality of objects having various shapes and various sizes and having various information described on the surface thereof. It may be. That is, the shape, size, and / or information described on the surface of each of the plurality of objects imaged by the image processing system 1 may be different. The image processing system 1 can extract only objects of a predetermined category from a plurality of objects.

Further, the information terminal 2 may be a mobile communication terminal, a smartphone, a notebook computer, and / or a tablet-type PC, and may be an information terminal such as a PC or a clock that can be connected to an imaging device capable of capturing moving images. Good. Further, the communication network 4 is a communication network such as a mobile phone network and / or the Internet. The communication network 4 may also include a communication network such as a wired LAN and a wireless LAN. The details of the image processing system 1 according to the present embodiment will be described below, but the names, numerical values, quantities, etc. in the above description and the following description are merely examples, and are limited to these names, numerical values, quantities, etc. I add that there is no such thing.

[Details of the configuration of the image processing system 1]
FIG. 2 shows an example of the functional configuration of the image processing system according to the embodiment of the present invention. In the following description, an example in which the object to be extracted is a receipt will be mainly described.

<Outline of the configuration of the image processing system 1>
The image processing system 1 includes a moving image imaging unit 10 that captures a moving image, a constituent image extracting unit 12 that extracts a moving image constituent image from the moving image, a resizing processing unit 14 that resizes the image, and a predetermined location of a predetermined object from the image. A coordinate acquisition unit 16 for acquiring coordinates, an image area extraction unit 18 for extracting an image area, an image processing unit 20 for performing predetermined processing on an image, a direction adjustment unit 22 for adjusting the direction of an image object, and predetermined An information storage unit 24 that stores the information of the above, a learning model generation unit 26 that generates a learning model, an input unit 28 that accepts input of predetermined information, an output unit 30 that outputs predetermined information, and text on the surface of the object. It includes a reading unit 32 for reading data and the like.

The image processing system 1 may not only have the plurality of components at physically the same location, but may also install some of the plurality of components at physically separated positions. For example, the image processing system 1 may be configured only by the information terminal 2 (that is, may be configured only locally), or is connected to the information terminal 2 and the information terminal 2 by a communication network 4 or the like. It may be configured to include a server 3. When the image processing system 1 is configured to include the information terminal 2 and the server 3, even if the information terminal 2 includes a part of the plurality of components and the server 3 includes the remaining components. Good. In this case, for example, the information terminal 2 can capture a moving image and supply the moving image to the server 3 to execute a predetermined process. The server may be an aggregate of a plurality of servers, and in this case, each server bears a part or all of other components except the moving image capturing unit 10. For example, a part of a plurality of processes of the image processing system 1 is executed on the information terminal 2 (for example, the information terminal 2 executes from imaging of a moving image to generation of a resized image), and other processes are executed on another one or more servers. (For example, the processing after the generation of the resized image is executed on the server). Further, the image processing system 1 may be composed of an imaging device having a moving image imaging unit 10 and one or more image processing devices having other components other than the moving image capturing unit 10. When "one or more image processing devices" are composed of a plurality of processing devices, other components other than the moving image imaging unit 10 can be appropriately allocated to each processing device according to the information processing capacity and the function to be fulfilled.

<Details of the configuration of the image processing system 1>
(Video imaging unit 10, constituent image extraction unit 12)
The moving image capturing unit 10 captures a moving image in the imaging region. The moving image imaging unit 10 captures one or more objects included in the imaging region as a moving image. The moving image capturing unit 10 can take an image of the image pickup target from directly above (that is, a depression angle of 90 degrees), or can take an image from a depression angle of less than 90 degrees. The moving image capturing unit 10 may take an image of a moving image by appropriately adjusting the frame rate. The moving image imaging unit 10 supplies the captured moving image to the constituent image extraction unit 12. The constituent image extraction unit 12 extracts a moving image constituent image from the moving image received from the moving image capturing unit 10. The constituent image extraction unit 12 extracts a plurality of moving image constituent images from the moving image. Here, the moving image is an image of various formats constituting a frame image, a field image, and other moving images. The constituent image extraction unit 12 supplies the extracted moving image constituent image to the resizing processing unit 14 and the image area extraction unit 18.

(Resize processing unit 14)
The resizing unit 14 resizes the image to generate a resized image. Specifically, the resizing processing unit 14 resizes the moving image constituent image extracted by the constituent image extracting unit 12 to generate a resized image. For example, the resizing processing unit 14 generates a resized image in which the size of the moving image constituent image is reduced. In this case, the resizing unit 14 may transform the rectangular moving image constituent image into a square resizing image. The resizing processing unit 14 generates, for example, a resized image obtained by resizing a moving image having a size of 3000 px × 2000 px in vertical and horizontal pixels to a size of 300 px × 300 px in vertical and horizontal pixels. The processing speed can be improved by performing the resizing process by the resizing processing unit 14. The resizing unit 14 supplies the resizing image to the coordinate acquisition unit 16.

(Coordinate acquisition unit 16)
The coordinate acquisition unit 16 acquires the coordinates of a predetermined position of the object based on the moving image of the moving image of the object. The coordinate acquisition unit 16 acquires the coordinates of a predetermined position of the object to be extracted from the image by using a learning model prepared in advance, which will be described later. The coordinates of the predetermined location are one or more rectangular regions centered on the corners of the object, and the length from the center to the outer edge of the moving image constituent image or the outer edge of the generated image generated based on the moving image constituent image is the shortest. The side whose vertical line is the straight line that becomes the distance is the coordinate of the center when a rectangular region having a size in contact with the outer edge of the moving image or the outer edge of the generated image is formed. Specifically, the coordinate acquisition unit 16 acquires the coordinates of a predetermined position of the object from the resize image as the generated image received from the resize processing unit 14. The coordinates of the predetermined location are the coordinates of the characteristic part of the object, for example, when the object is rectangular, the coordinates of the four corners or the coordinates of at least a part of the corners. That is, the coordinates of the predetermined location are perpendicular lines of one or more rectangular regions (for example, squares) centered on the corners of the object, and the length from the center to the outer edge of the resized image is the shortest distance. It is the coordinate of the center when a rectangular area having a size in which the side touches the outer edge of the resized image is formed. When the coordinate acquisition unit 16 acquires the coordinates of a part of the predetermined portion of the object from the resized image, the coordinate acquisition unit 16 estimates the coordinates of the remaining predetermined portion based on the learning model. As an example, when the object is rectangular, the coordinate acquisition unit 16 acquires the coordinates of the three corners of the object and estimates the coordinates of the remaining one corner using the coordinates of the three corners. Here, since the moving image is composed of a plurality of moving image constituent images, there are also a plurality of resizing images generated by the resizing processing unit 14. When a plurality of resized images are acquired from the resizing unit 14, the coordinate acquisition unit 16 selects a resized image capable of acquiring all the coordinates of a predetermined position of the object to be extracted, and acquires the coordinates from the selected resized image. You may. The coordinate acquisition unit 16 supplies information regarding the acquired coordinates to the image area extraction unit 18.

(Image area extraction unit 18)
The image area extraction unit 18 extracts an image area including an object to be extracted from the moving image constituent image based on the coordinates acquired by the coordinate acquisition unit 16. Specifically, the image area extraction unit 18 projects the coordinates of a predetermined position acquired from the resized image onto the moving image constituent image to extract the image area. That is, the image area extraction unit 18 projects the coordinates of a predetermined position of the object acquired from the resized image generated from the moving image constituent image onto the moving image constituent image before resizing, and the extraction target included in the moving image constituent image. Extract the image area that contains the object that is. In this case, the image area extraction unit 18 can add a predetermined margin area and extract the image area from the moving image. That is, the image area extraction unit 18 can extract an area including a predetermined margin area outside the area specified by the projection of the coordinates as an image area. The image area extraction unit 18 supplies the extracted image area to the resizing processing unit 14.

Then, the resizing processing unit 14 resizes the image area received from the image area extracting unit 18 again to generate a resized image area. That is, one resized image is generated from one moving image constituent image, and one image area is extracted from the one moving image constituent image using the coordinates acquired from the one resized image. Then, since the resized image area is generated by performing the resizing process on this one image area, the predetermined area (one image area) extracted from the one moving image constituent image is resized again. .. The resizing unit 14 supplies the resizing image area to the coordinate acquisition unit 16.

Subsequently, the coordinate acquisition unit 16 acquires the coordinates of a predetermined position of the object to be extracted based on the resized image area. That is, the coordinate acquisition unit 16 acquires the coordinates of a predetermined position of the object to be extracted from the resized image area as the generated image. The coordinates of the predetermined location are the coordinates of the characteristic part of the object, for example, when the object is rectangular, the coordinates of the four corners or the coordinates of at least a part of the corners. Specifically, the coordinates of the predetermined location are one or more rectangular regions centered on the corners of the object, and the sides whose perpendicular lines are straight lines having the shortest length from the center to the outer edge of the resized image region. These are the coordinates of the center when a rectangular area of a size tangent to the outer edge of the resized image area is formed. Also in this case, the coordinate acquisition unit 16 acquires the coordinates of a predetermined position of the object to be extracted from the resized image area by using the learning model described later. Further, when the coordinate acquisition unit 16 acquires the coordinates of a part of the predetermined portion of the object from the resized image area, the coordinate acquisition unit 16 estimates the coordinates of the remaining predetermined portion based on the learning model. As an example, when the object is rectangular, the coordinate acquisition unit 16 acquires the coordinates of the three corners of the object and estimates the coordinates of the remaining one corner using the coordinates of the three corners. The coordinate acquisition unit 16 supplies information regarding the acquired coordinates to the image area extraction unit 18. Then, the image area extraction unit 18 projects the coordinates of the predetermined portion acquired from the resized image area onto the moving image constituent image, and extracts the object image area of the object to be extracted. The image area extraction unit 18 supplies the extracted object image area to the image processing unit 20 and the information storage unit 24.

(Image processing unit 20)
The image processing unit 20 performs predetermined image processing (for example, correction processing such as blurring, distortion, rotation, etc.) on the image area extracted by the image area extraction unit 18 to generate an object image area. The image area extraction unit 18 may be used as an object image area without performing image processing by the image processing unit 20 on the extracted image area. The image processing unit 20 performs image processing on the object image area for the purpose of appropriately executing the reading and input of information in the reading unit 32 described later. For example, the image processing unit 20 sets the receipt at an oblique angle when the object image area is an image including the object having a shape deformed from the original shape of the predetermined object (for example, when the object is a receipt). When imaged from, the moving image includes an image of a receipt that is quadrilateral but not rectangular.), Transforms into a rectangular object by processing such as affine transformation. As a result, the image processing unit 20 removes key stoning, which is a phenomenon in which an object imaged from an oblique angle is included in the object image area as a trapezoidal object. Further, the image processing unit 20 can also perform image processing such as binarization processing and sharpness processing on the object image area for the purpose of causing the reading unit 32 to read a clearer image. The image processing unit 20 supplies the image after image processing to the direction adjusting unit 22.

(Direction adjusting unit 22)
The direction adjusting unit 22 adjusts the direction of a predetermined object included in the object image area. That is, the orientations of the predetermined objects included in the image processed by the image processing unit 20 are not always aligned with the predetermined directions. Therefore, the direction adjusting unit 22 executes a process of aligning the orientation of a predetermined object included in the image in a predetermined direction for the purpose of appropriately capturing / inputting information in the reading unit 32 described later. For example, when the orientation of the image changed to a rectangle by the affine transformation in the image processing unit 20 is based on the long side of the rectangle, as an example, the long side is 0 °, 90 ° with respect to the reference. It is conceivable to take four situations such as 180 ° and 270 °. The direction adjusting unit 22 views the direction of a predetermined object (for example, an object having a rectangular shape such as a receipt) included in the image obtained after image processing such as affine transformation in the image processing unit 20 as an example in a front view. The process of rotating the image in the vertically long direction (that is, the direction in which the short side is located in the horizontal direction and the long side is located in the vertical direction when the display unit of the information terminal 2 or the like is observed from the front). To execute. As a result, the direction adjusting unit 22 can align the direction of the predetermined object included in the object image area with the predetermined direction. The direction adjusting unit 22 supplies the object image area after adjusting the direction to the information storage unit 24 and the reading unit 32.

The direction adjusting unit 22 previously collects data of a plurality of predetermined objects (for example, image data of a rectangular object in which the short side of the rectangle is in the direction along the horizontal direction in front view). A learning model can also be generated by randomly classifying and learning a predetermined type of class according to a set rule. This learning model is a model that can be inferred to recognize the upward direction of a predetermined object as being upward. If the area of a predetermined object can be recognized in the upward direction, the area of a predetermined object can be easily transformed into a rectangle. Further, the learning model and the recognition method by Tesseract may be combined, and higher accuracy can be obtained by such a combination.

(Information storage unit 24)
The information storage unit 24 stores an object image area whose direction has been adjusted, that is, an object image area suitable for reading processing in the reading unit 32. The information storage unit 24 can store information such as the object image area, the imaging date of the moving image including the object image area, and the imaging time in association with the user ID that identifies the user, for example. Various information stored in the information storage unit 24 is stored in the information terminal 2 or an external server (for example, an external server of the image processing system 1 and used for accounting processing, accounting processing, etc.). Can be supplied. Further, the information terminal 2 does not have to have the information storage unit 24. In this case, the information storage unit 24 has an external server connected to the information terminal 2 so that bidirectional communication is possible via the communication network 4. You can do it.

(Learning model generation unit 26)
Based on the learning model, the coordinate acquisition unit 16 determines whether or not the resized image received from the resizing processing unit 14 includes a predetermined object, and determines the coordinates of at least a part of the predetermined object of the predetermined object. get. Further, the coordinate acquisition unit 16 acquires the coordinates of at least a part of the predetermined objects included in the resized image area received from the resizing processing unit 14 based on the learning model. The coordinate acquisition unit 16 can execute the determination by using a known image recognition technique or machine learning.

That is, the coordinate acquisition unit 16 determines whether or not the predetermined object is included in the moving image configuration image, the resized image, and / or the resized image area by using the learning model prepared by learning the features of the predetermined object in advance. To do. The coordinate acquisition unit 16 does not recognize an object that does not have the characteristics of a predetermined object as a predetermined object. The coordinate acquisition unit 16 performs inference processing using a learning model constructed by training a large number of images of a predetermined object or the like using a neural network, so that the predetermined object is a moving image, a resized image, and / or. It can be determined whether or not it is included in the resized image area. For example, the coordinate acquisition unit 16 uses the features in the image of the moving image, the resized image, and / or the object extracted from the resized image area and the learning model, and the moving image, the resized image, and / or It is determined whether or not a predetermined object exists in the resized image area, and whether or not each object included in the moving image constituent image, the resized image, and / or the resized image area is a predetermined object.

Specifically, the learning model generation unit 26 includes an image including a predetermined object, a background image in which a predetermined object can be arranged, and a combination of an image including the predetermined object and a background image in which the predetermined object can be arranged. When a moving image, which is an captured image, is input by learning by the gradient method as an example, the moving image constituent image and the resized image constituting the moving image, and / or the object included in the resized image area are predetermined. In order to identify whether or not it is an object, a learning model that outputs one or more rectangular areas centered on one or more corners of a predetermined object, coordinates of each center, and / or an image of the predetermined object is used. Generate. The learning model generation unit 26 uses the moving image captured by the moving image capturing unit 10 and / or the moving image acquired outside the image processing system 1 or in an image processing system 1 different from the image processing system 1 as teacher data. , You may generate a learning model.

More specifically, the learning model generation unit 26 uses an image including a predetermined object, a background image, or the like, and determines whether or not the object included in the image is a predetermined object by determining one or more of the predetermined objects. A learning model is generated in which a rectangular area (that is, a bounding box) centered on a location is extracted. The learning model generation unit 26 does not set the bounding box including the entire predetermined object to be extracted as the extraction target as in the conventional case, but the bounding box centered on one or more predetermined points of the predetermined object. Generate a learning model for which pairs are extracted. That is, the learning model generation unit 26 is not a learning model that outputs an area occupied by the image of the predetermined object based on one image including the entire predetermined object, but a plurality of predetermined locations of the predetermined object. Based on a plurality of bounding boxes centered on each, a learning model for outputting the area occupied by the image of the predetermined object is generated. For example, when an image containing one or more objects including a receipt is input, the learning model generation unit 26 is recognized by four bounding boxes centered on the four corners of the receipt and four bounding boxes. A learning model is generated in which the image of the receipt and / or the coordinates of the four corners are extracted. When an image containing one or more objects including a receipt is input, the learning model generation unit 26 recognizes it by one or more bounding boxes centered on a part of a corner of the receipt and one or more bounding boxes. A learning model may be generated in which the image of the receipt and / or the coordinates of one or more corners are extracted.

The learning model generation unit 26 may generate a learning model by expanding the data of an image including a predetermined object and artificially increasing the learning data for learning. For example, the learning model generation unit 26 includes not only an object image including a predetermined object but also a deformed image obtained by deforming the object image as an image including a predetermined object (the deformed image is, for example, a predetermined object. An image in which a part is missing, an image in which an object image is rotated by a predetermined angle, an image in which an object image is distorted, etc.), a noise image in which a predetermined noise is added to the object image, and a plurality of predetermined objects. An object image or the like including the object can be used. Further, the learning model generation unit 26 not only uses an image in which one predetermined object is captured from the front, but also uses an image in which the one predetermined object is captured from various angles, and the front. It is also possible to transform an image in a state taken from the above into an image in a state taken from various angles and use it. As an object image including a plurality of predetermined objects, an image in which one predetermined object is overlapped with another predetermined object, or a part of the predetermined objects protrudes out of the imaging area and is included in the imaging area. An image or the like containing only a part of the predetermined object can be used. Here, when a plurality of predetermined objects are included in the object image, any one predetermined object can be trained as a predetermined object to be recognized (for example, the object image includes a plurality of predetermined objects). If so, it is possible to learn to recognize a predetermined object located on the leftmost or rightmost as a predetermined object included in the object image.)

Further, the learning model generation unit 26 can also generate a learning model by superimposing various background images on an image of a predetermined object. As the background image, a wide variety of background images having different colors, brightness, brightness, contrast, and / or presence / absence of light reflection can be used. In other words, various situations are assumed in the environment where receipts are placed. For example, if a receipt is placed on the desk, if the desk is white or brown, or if the desk reflects fluorescent light depending on the indoor environment, carpets of various colors and surface shapes Various situations are assumed, such as when placed in. Therefore, the learning model generation unit 26 generates a learning model by superimposing various background images on the image of a predetermined object.

Then, the learning model generation unit 26 generates one or more predetermined locations when generating a learning model that outputs a region occupied by the image of the predetermined object based on a bounding box centered on the predetermined location of the predetermined object. Label each bounding box centered on.

FIG. 3 shows an outline of a method of labeling a predetermined object in the learning model generated by the learning model generation unit according to the present embodiment.

The learning model generation unit 26 according to the present embodiment acquires the coordinates of a predetermined position of the object included in the image, and has a rectangular region (that is, a bounding box, for example, a square) having the acquired coordinates as the center coordinates. Is formed, and the formed one or more rectangular regions are used as training data for identifying the object. By using this learning model, the coordinate acquisition unit 16 only needs to calculate the center of each rectangular area (bounding box) when acquiring the coordinates of the correct corner of the area occupied by the predetermined object. The corner position calculation can be facilitated.

That is, when the image 130 includes a predetermined object (for example, a receipt object 84), the learning model generation unit 26 is one or more rectangular regions centered on the coordinates of the corners of the predetermined object 84. Therefore, a rectangular region having a size in which the side whose vertical line is a straight line having the shortest length from the center to the outer edge of the image 130 including the predetermined object 84 is in contact with the outer edge of the image 130 is formed. For example, as shown in FIG. 3, when the image 130 includes the predetermined object 84, the learning model generation unit 26 has the coordinates of each of the four corners of the object 84 (that is, the coordinates 160, the coordinates 162, and the coordinates 164). , And a rectangular area centered on the coordinates 166) (that is, a rectangular area 170, a rectangular area 172, a rectangular area 174, and a rectangular area 176). In this case, the size of each rectangular area is defined by the distance from the coordinates of the center of each rectangular area to the outer edge of the image 130. For example, the rectangular area 170 is a square having a side 170a centered on the coordinates 160 at the corner of the object 84 and having a perpendicular line whose distance from the center to the outer edge of the image 130 is the shortest, which is in contact with the outer edge of the image 130. Is. Other rectangular areas are formed in the same manner. Then, the learning model generation unit 26 uses a large number of images of a predetermined object to be extracted and a large number of images obtained by superimposing an image of a predetermined object to be extracted on a background image as teacher data, as described above. A predetermined object in the image is identified based on the formed rectangular area and the coordinates of the center of the rectangular area, and one or more rectangular areas centered on one or more corners of the predetermined object, the center coordinates of each rectangular area. And / or generate a learning model to output an image of the given object.

The reason why the learning model generation unit 26 defines the width of each rectangular region as the distance in contact with the outer edge of the image 130 is that the present inventor has examined variously and found that one or more of them are centered on the coordinates of the corners of the object 84. A rectangular area having a side whose perpendicular line is a straight line having the shortest length from the center to the outer edge of the image 130 including the object 84 forms a rectangular area having a size in contact with the outer edge of the image 130 (that is,). , Maximizing the size of each rectangular area within the range of image 130 while keeping the rectangular shape square), the accuracy of determining whether one or more objects included in the image are predetermined objects. This is the result of obtaining the finding that

That is, unlike the conventional technique of identifying a predetermined object by using a rectangular area including the entire predetermined object, the learning model according to the present embodiment is for identifying the predetermined object and outputting an image of the object. Can be associated with a plurality of (for example, four) bounding box sets and a predetermined object category (for example, a receipt). Then, in the image processing system 1, when an image (for example, a moving image constituent image, a resized image, and / or a resized image area) is input, the inference processing based on the four bounding boxes is executed using the learning model. Then, it is possible to identify whether or not the object included in the image is a predetermined object, and to output the coordinates of a predetermined position of the object and the image area of the object.

There is no limitation on the method of detecting and forming the bounding box. For example, YOLO, Fast R-CNN, Single Shot Multi Detection (SSD) and the like can be used.

Then, the coordinate acquisition unit 16 determines whether or not the object included in the resized image is a predetermined object based on the learning model generated by the learning model generation unit 26, and if it is a predetermined object, the predetermined object. Gets the coordinates of a given location on an object. Further, the coordinate acquisition unit 16 acquires the coordinates of a predetermined position of a predetermined object included in the resized image area based on the learning model. Then, the coordinate acquisition unit 16 supplies the information regarding the acquired coordinates to the image area extraction unit 18.

(Reading unit 32)
The reading unit 32 reads various information written on the surface of the object included in the object image area whose direction is adjusted, which is received from the direction adjusting unit 22. The reading unit 32 uses, for example, Optical Character Recognition / Reader (OCR) or the like to read various types of information written on the surface of the object. As an example, when the object is a receipt, the information read by the reading unit 32 is information such as a date, an amount of money, and a telephone number. The reading unit 32 can store the read information in the information storage unit 24. The information storage unit 24 stores the read information together with the information regarding the imaging date and the imaging time of the object image area used for reading the information in association with the user ID, for example.

(Input unit 28)
The input unit 28 receives input of various information and predetermined instructions from the user. The input unit 28 is, for example, a touch panel, a keyboard, a mouse, a microphone, a gesture sensor, or the like of the information terminal 2. The input unit 28 supplies the predetermined instruction to a predetermined component of the image processing system 1. Each component that receives the predetermined instruction performs a predetermined function.

(Output unit 30)
The output unit 30 outputs various processing results executed in the image processing system 1. The output unit 30 outputs various processing results and stored information in a perceptible manner by the user. Specifically, the output unit 30 outputs various processing results and stored information as still images, moving images, sounds, texts, and / or physical phenomena such as vibration and light. For example, the output unit 30 is a display unit, a speaker, or the like of the information terminal 2.

[Processing flow of image processing system 1]
FIG. 4 shows an outline of the first step of the processing of the image processing system according to the present embodiment, FIG. 5 shows an outline of the reason for providing the margin region, and FIG. 6 shows the outline of the processing of the image processing system according to the present embodiment. The outline of the second step is shown. Further, FIG. 7 shows an outline of the overall processing flow of the image processing system according to the present embodiment.

First, as shown in FIG. 7, the learning model generation unit 26 uses a feature amount (for example, a set of four bounding boxes based on the coordinates of the corners of the object) of a predetermined object (for example, a receipt), or a part of the object. Acquired or generated, acquired or generated teacher data containing a combination of one or more (preferably two or more) bounding box sets based on the coordinates of the corners of a given object (eg, receipt). Based on the teacher data, input the resized image or the resized image area, and input one or more rectangular areas centered on one or more corners of the predetermined object included in the resized image or the resized image area and / or the image of the predetermined object. A learning model to be output is generated (step 10. Hereinafter, the step is referred to as "S").

Then, for example, the camera as the moving image capturing unit 10 of the information terminal 2 images the moving image 110 of a plurality of objects (a predetermined object and / or another object different from the predetermined object) (S12). As an example, as shown in FIG. 4A, the moving image capturing unit 10 images the moving image 110 of the object 86 (for example, a receipt). In the example of FIG. 4A, it is shown that the moving image 110 is composed of a plurality of moving image constituent images (for example, the moving image constituent image 120a, the moving image constituent image 120b, the moving image constituent image 120c, etc.). In this case, the moving image captured by the moving image capturing unit 10 may be a moving image in which a plurality of objects are arranged on a plane or a moving image in which a plurality of objects are turned over one by one. Good. Further, in a state where a plurality of objects are arranged on a plane, the directions of the objects may not be the same, and another object may overlap a part of one object. Further, the moving image imaging unit 10 may move the imaging region in the horizontal direction or the vertical direction. In addition, there is no limit to the size of the moving image.

Next, the constituent image extraction unit 12 converts the moving image captured by the moving image capturing unit 10 and extracts a plurality of moving image constituent images (S14). Then, the resizing processing unit 14 performs resizing processing on the extracted plurality of moving image constituent images to generate a resizing image (S16). For example, as shown in FIG. 4B, the resizing processing unit 14 resizes the moving image constituent image 120a to resize the moving image 140a, the moving image constituent image 120b to the resized image 140b, and the moving image constituent image 120c to be resized. Generate 140c.

Subsequently, the coordinate acquisition unit 16 uses the learning model 260 generated in advance by the learning model generation unit 26 to determine whether or not a predetermined object (for example, a receipt) is included in the resized image, and the predetermined object is If it is included, the coordinates of one or more corners (typically four corners) of the predetermined object in the resized image are acquired (S18). Here, the coordinate acquisition unit 16 determines whether or not a predetermined number of coordinates can be acquired when acquiring the coordinates of a predetermined object (S20). For example, if a given object is a rectangular receipt, the coordinate acquisition unit 16 may use the coordinates (ie, four coordinates) of the four corners of one given object or some corners (typically two). Determine if the coordinates of one or more corners) can be obtained. When the coordinate acquisition unit 16 determines that a predetermined number of coordinates cannot be acquired (No in S20), the configuration image extraction unit 12 extracts another video configuration image from the video 110 (S14). On the other hand, when the coordinate acquisition unit 16 determines that a predetermined number of coordinates can be acquired (Yes in S20), the coordinate acquisition unit 16 acquires a predetermined number of coordinates. Here, when the coordinates acquisition unit 16 acquires the coordinates of a part of the corners of the predetermined object in the resized image, the coordinates acquisition unit 16 estimates and acquires the coordinates of the remaining corners using the learning model.

For example, as shown in FIG. 4C, the resized image 140a and the resized image 140b include only a part of a predetermined object, and a part of the four corners of the object is included in the resized image 140a and the resized image 140b. Is not included. Therefore, the coordinate acquisition unit 16 determines that the coordinates of the predetermined location of the predetermined object cannot be acquired from the resized image 140a and the resized image 140b. On the other hand, the resized image 140c includes the entire predetermined object 86. Therefore, the coordinate acquisition unit 16 determines that the coordinates (that is, the coordinates 160a, the coordinates 162a, the coordinates 164a, and the coordinates 166a) of the predetermined positions (that is, the four corners) of the object 86 can be acquired from the resized image 140c, and these Get the coordinates.

Then, as shown in FIG. 4D, the image area extraction unit 18 projects the coordinates acquired by the coordinate acquisition unit 16 onto the moving image constituent image 120c (that is, the moving image constituent image before the resized image 140c is resized). (S22), the region of the object 86 of the moving image configuration image 120c is specified. Further, as shown in FIG. 4E, the image area extraction unit 18 acquires an image area 144 including the object 86 (S24). Here, the image area extraction unit 18 acquires an area including a predetermined margin area 180 around the object 86 as the image area 144. The reason for this will be described with reference to FIG.

First, as shown in FIG. 5A, it is assumed that the moving image configuration image 120 includes the object 88. When the resizing processing unit 14 resizes the moving image configuration image 120, the resizing image 142 is generated as shown in FIG. 5 (b). Then, the coordinate acquisition unit 16 acquires the coordinates of the four corners of the object 88 (that is, the coordinates 160b, the coordinates 162b, the coordinates 164b, and the coordinates 166b) from the resized image 142 using the learning model. Subsequently, the image area extraction unit 18 projects the coordinates acquired by the coordinate acquisition unit 16 onto the moving image configuration image 120 to acquire the image area.

In this case, since the coordinates acquired from the resized image 142 are projected onto the moving image configuration image 120 having a large image size before resizing, the position of each coordinate may deviate from the actual position. As an example, it is assumed that the number of vertical and horizontal pixels of the moving image constituent image 120 is 3840px × 2160px, and the number of vertical and horizontal pixels of the resized image 142 which is resized is 300px × 300px. In this case, there is a difference of 7.2 times in the horizontal direction and 12.8 times in the vertical direction with respect to the resized image 142 as a reference between the size of the resized image 142 and the size of the moving image constituent image 120. Therefore, when the coordinates acquired from the resized image 142 are projected onto the moving image configuration image 120, the coordinate positions may deviate from the actual coordinate positions. For example, when the coordinates 160b of the resized image 142 are projected onto the moving image constituent image 120, the coordinates 160c may be deviated in predetermined pixel units as shown by black circles in FIG. 5 (c). The same applies to the other coordinates (coordinates 162c, coordinates 164c, and coordinates 166c). As a result, the rectangular image area defined by the image area extraction unit 18 projecting the coordinates acquired by the coordinate acquisition unit 16 onto the moving image configuration image 120 is an image area 144a (FIG. 5 (FIG. 5)) as shown in FIG. It may be defined as the area defined by the dotted line in c) or the image area 144b (the area defined by the alternate long and short dash line in FIG. 5C), and may deviate from the image area of the actual object 88. .. Therefore, when the image area extraction unit 18 projects the coordinates acquired by the coordinate acquisition unit 16 onto the moving image constituent image and specifies the area of the object of the moving image constituent image, the image area extraction unit 18 includes a predetermined margin area around the object. Acquire the area as an image area (that is, extract the coarse, image area). The size of the margin region may be determined, for example, according to the reduction magnification of the resizing by the resizing processing unit 14, the ratio of the size of the moving image constituent image to the size of the resized image, and the like.

Subsequently, when the image area extraction unit 18 has not acquired the image area a predetermined number of times (for example, twice) (No in S26), the resizing processing unit 14 uses the image area 144 extracted by the image area extraction unit 18. Resize to generate a resized image area (S16). That is, the resized image area is generated by using the image area 144 obtained in the first step. For example, the resizing unit 14 resizes the image area 144 including the margin area 180 around the object 86 shown in FIG. 6A to generate the resized image area 146 shown in FIG. 6B. The size of the resized image area is not limited, but may be, for example, a size in which the number of vertical and horizontal pixels is 300 px × 300 px.

Subsequently, the coordinate acquisition unit 16 acquires the coordinates of one or more corners (typically four corners) of a predetermined object included in the resized image area by using the learning model 260 (S18). Here, the coordinate acquisition unit 16 determines whether or not a predetermined number of coordinates can be acquired when acquiring the coordinates of a predetermined object (S20). However, since S20 has already passed once, the coordinate acquisition unit 16 may omit the determination of whether or not a predetermined number of coordinates can be acquired and acquire a predetermined number of coordinates.

For example, as shown in FIG. 6C, the coordinate acquisition unit 16 uses the learning model 260 and has four rectangular regions centered on the four corners of the object 86, and the outer edge of the resized image region 146 from each center. Coordinates of four centers (that is, coordinates 160d, coordinates 162d, coordinates) when four rectangular areas having a size in which the side whose vertical line is the straight line whose length is the shortest distance is in contact with the outer edge of the resized image area 146 are formed. 164d and coordinates 166d) are acquired. When the coordinate acquisition unit 16 acquires the coordinates of a part of the corners of the predetermined object in the resized image area 146, the coordinate acquisition unit 16 estimates and acquires the coordinates of the remaining corners using the learning model.

Then, as shown in FIG. 6D, the image area extraction unit 18 uses the coordinates acquired by the coordinate acquisition unit 16 as the moving image configuration image 120c (that is, the moving image configuration including the image area 144 of the resizing source of the resizing image area 146). It is projected onto the image 120c) (S22), and the image area of the object 86 of the moving image configuration image 120c is acquired (S24). Since the coordinate acquisition unit 16 has already acquired the coordinates of the four corners of the object 86 using the image area roughly extracted in the first step, even if the coordinates are projected on the original moving image configuration image 120c, It is possible to reduce the deviation from the coordinates of the actual four corners of the object 86 included in the original moving image configuration image 120c.

Then, since the image area extraction unit 18 has acquired the image area a predetermined number of times (for example, twice) (Yes in S26), the image processing unit 20 performs a predetermined image processing on the acquired image area (S28). As a result, the image area extraction unit 18 extracts the object image area 148 (S30). The image area extraction unit 18 stores the extracted object image area 148 in, for example, the information storage unit 24.

[Image processing program]
Each component of the image processing system 1 according to the present embodiment shown in FIGS. 1 to 7 has a program (that is, an image processing program) in an arithmetic processing unit such as a central processing unit (CPU). It can be achieved by executing it, that is, by processing it by software. It can also be realized by writing a program in advance in hardware as an electronic component such as an integrated circuit (IC). It is also possible to use software and hardware together.

The image processing program according to this embodiment can be incorporated in advance into, for example, an IC or a ROM. Further, the image processing program is a file in an installable format or an executable format, which is recorded on a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, or a semiconductor recording medium, and provided as a computer program. You can also do it. The recording medium in which the program is stored may be a non-transient recording medium such as a CD-ROM or a DVD. Further, the image processing program can be stored in advance in a computer connected to a communication network such as the Internet so that the image processing program can be provided by download via the communication network.

The image processing program according to the present embodiment works on the CPU and the like to transfer the image processing program to the moving image imaging unit 10, the constituent image extraction unit 12, the resizing processing unit 14, the coordinate acquisition unit 16, and the image processing unit 16 described in FIGS. It functions as an image area extraction unit 18, an image processing unit 20, a direction adjustment unit 22, an information storage unit 24, a learning model generation unit 26, an input unit 28, an output unit 30, and a reading unit 32.

(Effect of embodiment)
The image processing system 1 according to the present embodiment learns by using a data set in which one or more rectangular areas centered on the coordinates of a predetermined location of a predetermined object to be extracted are labeled on the predetermined object. Use the constructed learning model. Then, the image processing system 1 extracts a moving image constituent image from the moving image, generates a resized image from the extracted moving image constituent image, and acquires the coordinates of a predetermined position of a predetermined object to be extracted from the resized image using a learning model. Then, the acquired coordinates are projected onto the moving image constituent image to extract the image area including the object to be extracted, and this image area is resized to generate a resized image area, and the learning model is used. The coordinates of a predetermined part of a predetermined object to be extracted are acquired from the resized image area, and the acquired coordinates are projected onto the moving image constituent image to extract the object image area of the object to be extracted. Extract the target object image area. As a result, according to the image processing system 1, for example, an object that is a high-precision, high-speed, and appropriate extraction target can be extracted simply by capturing a moving image of a plurality of objects arranged unorganized on a desk or the like. An image of (for example, a receipt) can be extracted and stored in the information storage unit 24 as data for acquiring various information described in the object or data for image processing.

Further, for example, a case where the object to be extracted is a receipt will be taken as an example. In this case, the business card and the receipt to be extracted by the conventional technique are characterized in that wrinkles are more likely to occur on the receipt and there are many bends, and the receipt is thinner than the business card and the desk. It is difficult to recognize the edge when placed on the top. For example, if the edge of the receipt is difficult to detect due to the color relationship between the background and the receipt (for example, if the color of the receipt is white and the color of the desk that is the background is white), the edge is used in the prior art. It cannot be detected properly and the area of the object cannot be extracted from the video. Furthermore, unlike business cards, receipts have various aspect ratio variations. In this case, in the technique premised on edge detection as in the conventional technique, when the receipt is wavy or bent (for example, the object 82 shown in FIG. 1B), one receipt is originally used. At some point, it is detected as multiple individual areas with the crease as the boundary. Further, for example, since the aspect ratio of a business card is substantially constant, it is easy to estimate the area of the business card in the video composition image using a bounding box, but the receipt has various variations in the aspect ratio. Therefore, in the conventional technique premised on edge detection, it is difficult to estimate the area of the receipt using the bounding box.

On the other hand, the image processing system 1 according to the present embodiment does not require edge detection, and can recognize whether or not the object is a receipt based on the coordinates of a predetermined position of the object and the bounding box centered on the coordinates. Therefore, even when a receipt in a wavy or bent state or a plurality of receipts having an inconsistent aspect ratio are imaged as a moving image, they can be appropriately recognized and detected as individual receipts.

Further, since the learning model 260 according to the present embodiment trains the images of the receipts having various aspect ratios, according to the image processing system 1, each of the plurality of receipts whose aspect ratios are not constant is received. Can be properly recognized as a receipt. Further, since the learning model 260 trains the coordinates of the four corners of the receipts of various formats and the bounding box including the four corners, according to the image processing system 1, for example, information is provided on one receipt. Even when a plurality of areas in which is displayed are printed and there is a large space between one area and the other area, it can be appropriately recognized as one receipt. Since the learning model 260 trains the receipt image by superimposing the receipt image on various background images as well as the receipt image, the contrast difference between the background and the receipt is according to the image processing system 1. Even if is small, the image of the receipt can be properly extracted.

Specifically, in the image processing system 1 according to the present embodiment, after preparing a learning model 260 in which a predetermined object to be extracted is a receipt, the moving image imaging unit 10 includes a receipt, a business card, and a smartphone. The area was imaged and tested. As a result, the image processing system 1 according to the present embodiment appropriately extracts the object image area of the receipt from the moving image constituent image in real time. On the other hand, the image processing system 1 did not recognize the business card and the smartphone as a receipt.

Although the embodiments of the present invention have been described above, the embodiments described above do not limit the invention according to the claims. It should also be noted that not all combinations of features described in the embodiments are essential to the means for solving the problems of the invention. Further, the technical elements of the above-described embodiments may be applied alone, or may be divided and applied to a plurality of parts such as a program component and a hardware component.

The image processing system according to the present embodiment can also be referred to in the following appendix which should not be confused with the scope of claims.
(Appendix 1)
A video imager that captures an object as a video,
A component image extraction unit that extracts a moving image of the moving image,
A resizing processing unit that resizes the moving image configuration image and generates a resized image,
A coordinate acquisition unit that acquires the coordinates of a predetermined position of the object from the resized image, and
It is provided with an image area extraction unit that projects the coordinates onto the moving image constituent image and extracts an image area including the object from the moving image constituent image.
The resizing unit resizes the image area to generate a resized image area.
The coordinate acquisition unit reacquires the coordinates of the predetermined position of the object from the resized image area.
An image processing system in which the image area extraction unit projects the re-acquired coordinates onto the moving image constituent image to extract the object image area of the object.

1 Image processing system 2 Information terminal 3 Server 4 Communication network 10 Video imaging unit 12 Configuration image extraction unit 14 Resize processing unit 16 Coordinate acquisition unit 18 Image area extraction unit 20 Image processing unit 22 Direction adjustment unit 24 Information storage unit 26 Learning model generation Part 28 Input part 30 Output part 32 Reading part 80, 82, 84, 86, 88 Object 90 Desk 100 Resized image 110 Video 120, 120a, 120b, 120c Video composition image 130 image 140a, 140b, 140c Resized image 142 Resized image 144 , 144a, 144b image area 146 resized image area 148 object image area 150, 152, 154, 156 corners 150a, 152a, 154a, 156a coordinates 160, 162, 164, 166 coordinates 160a, 162a, 164a, 166a coordinates 160b, 162b, 164b, 166b Coordinates 160c, 162c, 164c, 166c Coordinates 160d, 162d, 164d, 166d Coordinates 170, 172, 174, 176 Rectangular Area 170a Side 180 Margin Area 260 Learning Model

Claims (9)

A resizing processing unit that resizes the video composition image of the video that captured the object and generates a resized image,
A coordinate acquisition unit that acquires the coordinates of the corners of the object from the resized image, and
It is provided with an image area extraction unit that projects the coordinates acquired from the resized image onto the moving image constituent image and extracts an image area including the object from the moving image constituent image.
The resizing unit resizes the image area to generate a resized image area.
The coordinate acquisition unit reacquires the coordinates of the corner of the object from the resized image area.
The image area extraction unit projects the re- acquired coordinates onto the moving image constituent image to extract the object image area of the object .
The resized image has a side whose perpendicular line is a straight line whose coordinates are one or more rectangular regions centered on the corner of the object and whose length from the center to the outer edge of the resized image is the shortest distance. An image processing system that is the coordinates of the center when the rectangular region having a size tangent to the outer edge of the is formed . The image processing system according to claim 1, wherein the image area extraction unit extracts the image area to which a predetermined margin area is added from the moving image constituent image. The image processing system according to claim 1 or 2 , wherein the coordinate acquisition unit acquires the coordinates of the corner of the predetermined object by using a learning model prepared in advance. The image processing system according to any one of claims 1 to 3 , further comprising an image processing unit that performs predetermined image processing on the object image area. A coordinate acquisition unit that acquires the coordinates of a predetermined location of the object based on the moving image of the moving image of the object.
It includes an image area extraction unit that extracts an image area including the object from the moving image constituent image based on the coordinates.
The coordinate acquisition unit acquires the coordinates of the predetermined position of the object based on the image area.
The image area extraction unit projects the acquired coordinates onto the moving image constituent image to extract the object image area of the object.
The coordinates of the predetermined location are one or more rectangular regions centered on the corners of the object, from the center to the outer edge of the moving image constituent image or the outer edge of the generated image generated based on the moving image constituent image. An image processing system in which a side having a straight line having the shortest length as a perpendicular line is the coordinate of the center when the rectangular region having a size in contact with the outer edge of the moving image constituent image or the outer edge of the generated image is formed. An image processing method for image processing systems
A resizing process that resizes the video composition image of the video that captured the object and generates a resized image,
A coordinate acquisition process for acquiring the coordinates of the corners of the object from the resized image, and
An image area extraction step of projecting the coordinates acquired from the resized image onto the moving image constituent image and extracting an image area including the object from the moving image constituent image.
A step of resizing the image area to generate a resized image area, and
A step of reacquiring the coordinates of the corner of the object from the resized image area, and
A step of projecting the reacquired coordinates onto the moving image constituent image to extract an object image area of the object is provided .
The resizing image is a side whose perpendicular line is a straight line whose corner coordinates are one or more rectangular regions centered on the corner of the object and whose length from the center to the outer edge of the resizing image is the shortest distance. coordinate der Ru image processing method of the center in the case of forming the rectangular region size in contact with the outer edge of the. An image processing program for an image processing system
On the computer
Resize processing function that resizes the video composition image of the video that captured the object and generates a resized image,
A coordinate acquisition function that acquires the coordinates of the corners of the object from the resized image, and
An image area extraction function that projects the coordinates acquired from the resized image onto the moving image constituent image and extracts an image area including the object from the moving image constituent image.
A function to resize the image area to generate a resized image area, and
A function to reacquire the coordinates of the corner of the object from the resized image area, and
A function of projecting the re- acquired coordinates onto the moving image constituent image to extract an object image area of the object is realized .
The resizing image is a side whose perpendicular line is a straight line whose corner coordinates are one or more rectangular regions centered on the corner of the object and whose length from the center to the outer edge of the resizing image is the shortest distance. the center coordinates der Ru image processing program in the case of forming the rectangular region size in contact with the outer edge. A resizing processing unit that resizes the video composition image of the video that captured the object and generates a resized image,
A coordinate acquisition unit that acquires the coordinates of the corners of the object from the resized image, and
It is provided with an image area extraction unit that projects the coordinates acquired from the resized image onto the moving image constituent image and extracts an image area including the object from the moving image constituent image.
The resizing unit resizes the image area to generate a resized image area.
The coordinate acquisition unit reacquires the coordinates of the corner of the object from the resized image area.
The image area extraction unit projects the re- acquired coordinates onto the moving image constituent image to extract the object image area of the object .
The resizing image is a side whose perpendicular line is a straight line whose corner coordinates are one or more rectangular regions centered on the corner of the object and whose length from the center to the outer edge of the resizing image is the shortest distance. An image processing server that is the coordinates of the center when the rectangular region having a size tangent to the outer edge of the is formed . When the captured image is input, one or more rectangular areas centered on one or more corners of the predetermined object are output in order to identify whether or not the object included in the captured image is a predetermined object. It ’s a learning model that makes the processor work.
In the learning model, an image including the predetermined object, a background image in which the predetermined object can be arranged, and a combination of the image including the predetermined object and the background image are learned as teacher data.
In the learning, a straight line having one or more rectangular regions centered on the corners of the predetermined object and having the shortest length from the center to the outer edge of the image including the predetermined object is defined as a perpendicular line. A learning model for forming the rectangular region whose sides are tangent to the outer edge of the image and identifying the predetermined object in the image using the formed rectangular region and the coordinates of the center of the rectangular region. ..

Images