WO2021261016A1 - Image processing device, image processing system, image processing program, and image processing method - Google Patents

Abstract

The present invention enables a subject imaged in a captured image to undergo perspective transformation on the basis of the captured image.　An image processing device (2) according to the present invention performs perspective transformation of a subject having a feature of a rectangular shape in a captured image in which the subject is imaged in a state in which perspective projection has been performed, and corrects the shape of the subject imaged in the captured image. The image processing device is characterized by being formed to have: an image acquisition unit (231) for acquiring a captured image; a reference specification unit (232) for specifying, from the subject imaged in the captured image, four reference points that form a square region and the respective coordinates of the four reference points; a storage unit (22) for storing a plurality of conversion patterns for perspective transformation of the subject; a pattern selection unit (233) for selecting a conversion pattern with which to perform perspective transformation of the subject from among the plurality of conversion patterns; and a correction unit (235) for correcting the shape of the subject imaged in the captured image on the basis of the conversion pattern.

Description

Image processing device, image processing system, image processing program, and image processing method

The present invention relates to an image processing device, an image processing system, an image processing program, and an image processing method.

Surveillance cameras that monitor the movement and entry / exit of goods and people are widely used indoors in stores such as supermarkets, home improvement stores, and convenience stores, and outdoors such as roads, shopping mall passages, and major roads. Generally, a surveillance camera is fixed at a predetermined position and captures a subject such as a product or a person within an angle of view. The captured image in which the subject is captured is analyzed by image recognition processing using AI (Artificial Intelligence).

In recent years, with the increase in resolution of image pickup devices such as digital cameras, a technique for capturing a wide range of subjects with one image pickup device equipped with a wide-angle lens has also been used for surveillance cameras. In addition, the surveillance camera is usually arranged above the subject so as not to interfere with the movement of the subject, and captures the subject from diagonally above. Therefore, the subject captured in the captured image is captured in a perspective-projected state depending on the positional relationship between the surveillance camera and the subject, such as the angle of view and orientation of the surveillance camera and the distance to the subject. That is, in the captured image, the subject is imaged in a distorted shape according to the above-mentioned positional relationship.

As described above, the captured image is analyzed by image recognition processing using AI. The image processing technique requires teacher data for AI to learn. However, when the teacher data is created in consideration of the above-mentioned distortion, the number of teacher data becomes enormous, the time for creating the teacher data increases, and the learning time for AI also increases. As a result, the development period and development cost increase.

So far, a technique for correcting the shape of a subject by performing fluoroscopic transformation of the subject captured in the captured image has been proposed (see, for example, Patent Documents 1 and 2).

In the technique disclosed in Patent Document 1, the parameters of perspective conversion are calculated using a reference medium (an image taken by capturing the reference medium) with marks at four corners in advance, and distortion correction data is obtained based on the parameters. Is created, and the distortion of another object to be processed (an image taken by another subject) is corrected based on the distortion correction data. As a result, the technology can reduce the amount of data processing during distortion correction. As described above, the technique employs a method of acquiring distortion correction data using a reference medium marked with a mark, that is, a method of executing calibration. Therefore, in this technology, a reference medium must be prepared for each subject.

The technique disclosed in Patent Document 2 estimates the position / orientation of the image pickup device relative to the paper surface from the four vertices of the rectangular paper surface captured in the captured image, and has a rectangular shape based on the estimated position / orientation. Four vertices in the three-dimensional space of the paper are estimated, the parameters of the fluoroscopic conversion are calculated based on the estimated positions of the four vertices, and the distortion of the captured image is corrected based on the parameters. As a result, the technology can perform fluoroscopic transformation of the subject captured in the captured image and correct the distortion of the subject even if the position and orientation of the image pickup device are unknown. As described above, the technique estimates the position and orientation of the image pickup device from the captured image, and calculates the parameters of the perspective conversion based on the estimated position and posture of the image pickup device. That is, the technique requires a process of estimating the position / orientation of the image pickup device for the captured image whose position / orientation of the image pickup device is unknown.

Japanese Unexamined Patent Publication No. 2002-197452 Japanese Unexamined Patent Publication No. 2004-040395

An object of the present invention is to perform fluoroscopic transformation of a subject captured in a captured image based on a captured image without requiring estimation of a reference medium or a position / posture of an imaging device.

The image processing apparatus according to the present invention performs fluoroscopic conversion of a subject to an image captured in a state in which a subject having a rectangular feature is perspectively projected, and corrects the shape of the subject captured in the captured image. An image processing device that acquires an image acquisition unit, four reference points that form a square region from the subject captured in the captured image, and a reference that specifies the coordinates of each of the four reference points. A specific part, a storage unit that stores a plurality of conversion patterns for see-through conversion of the subject, and a conversion pattern for see-through conversion of the subject are selected from a plurality of conversion patterns stored in the storage unit based on the coordinates. It is characterized by having a pattern selection unit and a correction unit that corrects the shape of the subject captured in the captured image based on the conversion pattern.

According to the present invention, the subject captured in the captured image can be fluoroscopically transformed based on the captured image.

It is a schematic diagram which shows the example of the arrangement of the pixel in the captured image. It is a network block diagram which shows the embodiment of the image processing system which concerns on this invention. It is a functional block diagram of the image pickup apparatus provided in the image processing system of FIG. It is a functional block diagram which shows the embodiment of the image processing apparatus which concerns on this invention. It is a schematic diagram which shows the example of the captured image. It is a flowchart which shows the embodiment of the image processing method which concerns on this invention. It is a flowchart of the reference point specifying process included in the image processing method of FIG. It is a schematic diagram which shows the example of the reference point specified by the reference point specifying process of FIG. 7. It is a flow chart of a part of the correction processing included in the image processing method of FIG. It is a flowchart of another part of the correction process of FIG. It is a schematic diagram explaining the 1st diagonal conversion pattern selected by the correction process of FIG. It is a schematic diagram explaining the 2nd diagonal conversion pattern selected by the correction process of FIG. It is a schematic diagram explaining the 1st rotation conversion pattern selected by the correction process of FIG. It is a schematic diagram explaining the 2nd rotation conversion pattern selected by the correction process of FIG. It is a schematic diagram which shows the example of the correction image generated by the correction processing of FIG.

Hereinafter, embodiments of the image processing apparatus, the image processing system, the image processing program, and the image processing method according to the present invention will be described with reference to the drawings.

● Information processing system ●
First, an embodiment of the information processing system (hereinafter referred to as "the present system") according to the present invention will be described.

This system specifies the coordinates of each of the four reference points of the subject in the captured image captured while the subject is perspective-projected, and the conversion pattern of perspective conversion is based on the positional relationship of the four specified reference points. Is selected, and the shape of the subject captured in the captured image is corrected based on the selected conversion pattern. Specifically, this system selects a conversion pattern for perspective conversion of the subject based on the degree of deformation of the subject captured in the captured image, and determines the shape of the subject based on the selected conversion pattern. Converts from the perspective projection state to the front view state.

The "subject" is an object to be imaged by an image pickup device described later. The subject is an object in which the entire subject to be imaged is rectangular (for example, an object having a rectangular shape in one-way view), or an object in which a part of the subject to be imaged has a rectangular feature. That is, for example, the subject is a product shelf in which products are displayed in a supermarket or a home center, a book shelf in which books are displayed in a bookstore or a library, a building such as a building or an apartment house, a straight road or a sidewalk. ..

The subject may be a space as long as the four corners of the rectangle (points constituting the four corners of the rectangle) can be recognized. That is, for example, the subject is a space extracted in a rectangular shape by four articles (for example, a product shelf, a shelf board, a product, a price tag, a sign attached in a store, etc.) or a rectangle with four stars as four corners. It may be a part of an aerial photograph or a landscape photograph with four corners of the night sky extracted into a shape.

The "rectangular shape" may be a shape that can specify a reference point in image processing described later, and is not only a rectangle and a square having right angles at all corners, but also a shape similar to a square (for example, rounded corners and / or sides). Also includes the squares that are used.

"Perspective projection" is to project (draw) the same object when a person visually observes a three-dimensional object onto a two-dimensional virtual plane in the same shape as it was observed, that is, the same object by perspective method. Means to project on a two-dimensional virtual plane.

The "captured image" is an image captured by an imaging device described later, and is, for example, a still image or each frame of a moving image. In the present invention, a subject in a perspective-projected state is captured in the captured image. That is, the captured image includes an image of the subject. The details of the captured image will be described later.

The "reference point" is, for example, a point where a rectangular region is formed by being connected by straight lines that do not intersect each other when the subject is viewed from the front or a surface having features. That is, for example, the reference point is a part of the subject or a part of a sign (for example, a two-dimensional bar code, a figure, a symbol, etc.) attached to the subject (for example, the central part of the sign). In the present invention, the reference point includes not only the four corners of one rectangular object but also a part of a plurality of physically separated objects. That is, for example, the reference point is a shelf, a desk, a window, a wall, a passage, four corners of a product package, or four articles arranged so as to form a rectangular space (for example, a product shelf, a shelf board, a product). , Price tags, signs attached to the store, etc.).

"Coordinates" is information indicating the position of each pixel in the captured image. In the present embodiment, the origin of the coordinates is the upper left of the captured image, and in the arrangement of the plurality of pixels constituting the captured image, the column direction is the X direction and the row direction is the Y direction. The coordinates include an X coordinate value indicating a coordinate value in the X direction and a Y coordinate value indicating a coordinate value in the Y direction.

FIG. 1 is a schematic diagram showing an example of pixel arrangement in a captured image.
In the figure, some pixels are indicated by “□”, and the illustration of other pixels is omitted. The figure shows that the column direction of the captured image (the left-right direction of the paper surface in FIG. 1) is the X direction, and the row direction of the captured image (the vertical direction of the paper surface of FIG. 1) is the Y direction. The figure shows that the coordinate value of the origin (upper left) is (0,0) and the coordinate value of the end point (lower right) is (m, n) (m, n are integers of 0 or more). The figure shows that the X coordinate value increases as the pixel arrangement in the captured image moves to the right, and the Y coordinate value increases as the pixel arrangement moves downward. That is, in the figure, in the front view of the captured image, the X coordinate value in the leftmost column of the captured image is the minimum value of the X coordinate value, and the X coordinate value in the right end column is the maximum value of the X coordinate value. It is shown that the Y coordinate value of the uppermost row is the minimum value of the Y coordinate value, and the Y coordinate value of the lower end row is the maximum value of the Y coordinate value.

An embodiment described below is an image processing apparatus according to the present invention (hereinafter, “the present apparatus”) for an image taken on a product shelf in which products are displayed in a retail store such as a supermarket or a convenience store. The content of the present invention will be described by taking as an example a case where the image processing method according to the present invention (hereinafter referred to as “the present method”) described later is executed to correct the shapes of the product shelves and the products. .. That is, the product shelf and the product are examples of subjects in the present invention.

● Configuration of image processing system FIG. 2 is a network configuration diagram showing an embodiment of this system.

The system S includes a plurality of image pickup devices 1 (1), ..., 1 (n) (n is an integer) and the present device 2. In the following description, the plurality of image pickup devices 1 (1), ..., 1 (n) are referred to as "imaging" unless the respective image pickup devices 1 (1), ..., 1 (n) are particularly distinguished. Notated as "device 1 (1 to n)".

Each image pickup device 1 (1 to n) and this device 2 are connected via a network N (communication line) that uses a wired communication method or a wireless communication method. The network N is, for example, a communication network such as the Internet, a mobile communication network, a LAN (Local Area Network), a WAN (Wide Area Network), and a Wi-Fi (registered trademark).

● Configuration of the image pickup device FIG. 3 is a functional block diagram of the image pickup device 1.

The image pickup device 1 captures a subject (product shelf, product) and generates an captured image. The image pickup device 1 is, for example, a camera unit that captures still images of product shelves in a sales floor at predetermined time intervals. Each image pickup device 1 (1 to n) is, for example, in a position (for example,) in the sales floor where an overall image of the front portion (front surface) of the corresponding product shelf and the products displayed on the product shelf can be imaged. It is placed on the ceiling or wall of the sales floor). The image pickup apparatus 1 includes a communication unit 11, a storage unit 12, an image pickup unit 13, and a control unit 14. The image pickup device is an example of the image pickup unit in the present invention.

The image pickup device may be a terminal device having a camera function including an image pickup unit described later. That is, for example, the image pickup device may be a smart phone having a camera function, a mobile phone, a tablet terminal, a laptop computer, or a digital camera having a communication function.

Further, the image pickup device may be a camera unit that captures a moving image, or may be a camera unit that captures a still image and a moving image.

Further, the image pickup device may image a plurality of product shelves (products), or may image a part (product) of the product shelves.

The communication unit 11 communicates with the present device 2 via the network N. The communication unit 11 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 12 stores the captured image captured by the image pickup unit 13. The storage unit 12 is a storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The image pickup unit 13 captures a product shelf (product). The image pickup unit 13 is composed of, for example, a lens group, a shutter, an image pickup element, an A / D converter, and the like. The image pickup unit 13 images, for example, a product shelf (product) arranged within the image pickup range at a predetermined frame rate. The imaging conditions (frame rate, imaging range, resolution, etc.) of the imaging unit 13 are controlled by the control unit 14. The captured image captured by the image pickup unit 13 is stored in association with information indicating the position of the image pickup device 1 that captured the captured image (position information) and information indicating the time when the captured image was captured (time information). It is stored in the part 12.

The control unit 14 executes various information processing according to a program to control the operation of the entire image pickup apparatus 1. The control unit 14 includes processors such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), DSP (Digital Signal Processor), and integrated circuits such as ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array). , RAM (ReadAccessMemory), ROM (ReadOnlyMemory) and other semiconductor memories.

● Configuration of Image Processing Device FIG. 4 is a functional block diagram showing an embodiment of the device 2.

The present device 2 specifies the coordinates of the four reference points P1, P2, P3, P4 (see FIG. 8, the same shall apply hereinafter) of the subject in the captured image, and determines the positional relationship between the specified four reference points P1 to P4. A conversion pattern for fluoroscopy conversion is selected based on the above, and the shape of the subject captured in the captured image is corrected based on the selected conversion pattern. The present device 2 is, for example, a personal computer such as a server. The device 2 is located, for example, in a supermarket office.

In the present device 2, the information processing program according to the present invention (hereinafter referred to as "the present program") operates, and this program cooperates with the hardware resources of the present device 2 to realize the present method described later. That is, this program causes the computer to function as the present device 2.

The apparatus 2 includes a communication unit 21, a storage unit 22, a control unit 23, a display unit 24, and an operation unit 25.

The communication unit 21 communicates with the image pickup apparatus 1 via the network N. The communication unit 21 is, for example, a communication interface such as a communication module or a communication circuit.

The storage unit 22 stores information (for example, a conversion pattern described later) necessary for the apparatus 2 to execute the method described later. The storage unit 22 is a storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The control unit 23 executes various information processing according to the program to control the operation of the entire device 2. The control unit 23 is composed of, for example, a processor such as a CPU, MPU, and DSP, an integrated circuit such as an ASIC and an FPGA, and a semiconductor memory such as a RAM and a ROM.

The control unit 23 includes an image acquisition unit 231, a reference specifying unit 232, a pattern selection unit 233, a calculation unit 234, and a correction unit 235.

The image acquisition unit 231 acquires an captured image from the storage unit 22. The operation of the image acquisition unit 231 will be described later.

The image acquisition unit in the present invention may acquire an image captured from the image pickup device via the communication unit.

The reference specifying unit 232 specifies the coordinates of each of the four reference points P1 to P4 and the four reference points P1 to P4 from the subject (product shelf) captured in the captured image. In the present embodiment, the four reference points P1 to P4 are the four corners in the front view of the product shelf. The specific operation of the reference specifying unit 232 will be described later.

The pattern selection unit 233 selects a conversion pattern for fluoroscopic conversion of the subject based on the coordinates of the four reference points P1 to P4 of the subject specified by the reference identification unit 232. The specific operation of the pattern selection unit 233 and the conversion pattern will be described later.

The calculation unit 234 calculates the parameters of the perspective transformation (parameters corresponding to the conversion pattern) based on the coordinates of the four reference points P1 to P4 of the subject specified by the reference identification unit 232. The specific operation and parameters of the calculation unit 234 will be described later.

The correction unit 235 corrects the shape of the subject captured in the captured image based on the conversion pattern. The specific operation of the correction unit 235 will be described later.

The image acquisition unit, the reference specifying unit, the pattern selection unit, the calculation unit, and the correction unit may be configured by a common processor or integrated circuit, a semiconductor memory, or an individual processor or integrated circuit. , May be configured by a semiconductor memory.

The display unit 24 is a device (for example, a liquid crystal display) that displays an captured image.

The operation unit 25 is a device (for example, a keyboard, a mouse, a touch panel) that accepts the user's operation so that the user of the apparatus 2 can select an arbitrary position with respect to the captured image displayed on the display unit 24. be. In the present embodiment, the display unit 24 and the operation unit 25 are, for example, a touch panel display.

● Image processing method ●
Next, for the operation of the present device 2, that is, for the present method, the captured images captured by the respective image pickup devices 1 (1 to n) are transmitted to the present device 2 via the network N, and the storage unit 22 of the present device 2 is transmitted. If it is stored in, it will be explained as an example. At this time, the position information and the time information associated with each captured image are also associated with the corresponding captured image and stored in the storage unit 22.

FIG. 5 is a schematic diagram showing an example of a captured image.
In this figure, the entire image of one product shelf Sa, a part of two product shelves Sb and Sc adjacent to the product shelf Sa, and the products displayed on each product shelf Sa to Sc are shown. Indicates that the image has been taken. That is, in the present embodiment, the captured image includes the images of the product shelves Sa to Sc and the images of the products. In the following description, each image included in the captured image (subject captured in the captured image) is represented by the name of the subject (product shelf Sa to Sc, product, etc.).

The figure shows that the captured image is taken from the diagonally left front of the product shelf Sa (diagonally to the right when viewed from the product shelf Sa). Further, the figure shows a state in which the product shelves Sa to Sc and the product are perspectively projected from the left side to the right side of the captured image so that the size in the Y direction (vertical direction on the paper surface in FIG. 5) becomes smaller. Indicates that the image is being taken. Further, the figure shows that the markers M1, M2, M3, and M4 are attached to the four corners of the product shelf Sa.

FIG. 6 is a flowchart showing an embodiment of the present method.
This method includes a reference point specifying process (S100) and a correction process (S200).

● Reference point specifying process First, the reference point specifying process (S100) will be described with reference to FIGS. 4 and 5.

The "reference point specifying process (S100)" is a process for specifying four reference points P1 to P4 for each specific imaging device described later.

FIG. 7 is a flowchart of the reference point specifying process (S100).

First, the image acquisition unit 231 acquires one captured image from the storage unit 22 as a reference specific image (S101). In the following description, the reference specific image is assumed to be the captured image shown in FIG. That is, in the reference specific image, four signs M1 to M4 are attached to the product shelves.

Next, the image acquisition unit 231 identifies the image pickup device 1 that has captured the reference specific image as the specific image pickup device based on the position information associated with the reference specific image (S102). Next, the image acquisition unit 231 determines whether or not the reference points P1 to P4 are specified in another captured image (past captured image) captured by the specific imaging device (S103).

When the reference points P1 to P4 are not specified in other captured images captured by the specific imaging device (“N” in S103), the reference specifying unit 232 is a reference specifying image based on the four labels M1 to M4. Four reference points P1 to P4 are specified from the subject imaged in (S104). Specifically, the reference specifying unit 232 executes known image processing (for example, pattern matching) on the reference specifying image, and detects the four labels M1 to M4. Next, the reference specifying unit 232 calculates, for example, a pixel corresponding to the center point of each of the labels M1 to M4, and specifies the pixel as the reference points P1 to P4.

FIG. 8 is a schematic diagram showing an example of reference points P1 to P4 specified by the reference specifying unit 232. The figure shows that the center point (pixel) of each of the markers M1 to M4 is specified as the reference point P1 to P4 in the reference specific image. The figure shows that in the reference specific image, the reference point P1 is arranged in the upper left, the reference point P2 is arranged in the lower left, the reference point P3 is arranged in the lower right, and the reference point P4 is arranged in the upper right. ..

The reference point specified by the reference specifying unit in the present invention is not limited to the center point of the sign. That is, for example, if the sign has a rectangular shape, the reference specifying unit may specify one pixel (for example, the lower right pixel) among the pixels corresponding to the four corners of the sign as a reference point.

Further, the reference specifying unit in the present invention may specify a pixel corresponding to a part of the subject as a reference point by using known image processing instead of a sign. That is, for example, the reference specifying unit detects the edge of the product shelf using a known image processing technique, and specifies the pixels corresponding to the four corners (upper left, lower left, upper right, lower right) of the product shelf as reference points. You may.

Furthermore, the shape of the sign is not limited to a rectangular frame as long as the pixel (or a group of pixels) serving as a reference point can be specified. That is, for example, the shape of the sign may be a "○" shape, a "+" shape, or an "X" shape.

Return to FIG.
Next, the reference specifying unit 232 specifies the coordinates (X coordinate value, Y coordinate value) of each of the reference points P1 to P4 (S105). The reference points P1 to P4 and the coordinates are associated with the specific image pickup apparatus and the reference specific image, and are stored in the storage unit 22 (S106).

Next, the reference specifying unit 232 specifies the relative positional relationship between the reference points P1 to P4 based on the coordinates of the reference points P1 to P4 (S107). Specifically, the reference specifying unit 232 identifies, for example, two reference points P1 and P2 having a small X coordinate value among the reference points P1 to P4, and the Y coordinate value among the reference points P1 and P2. The reference point P1 having a small value is specified as the first reference point, and the reference point having a large Y coordinate value is specified as the second reference point. Further, the reference specifying unit 232 identifies, for example, two reference points P3 and P4 having a large X coordinate value among the reference points P1 to P4, and a reference having a large Y coordinate value among the reference points P3 and P4. The point P3 is specified as the third reference point, and the reference point P4 having a small Y coordinate value is specified as the fourth reference point. That is, for example, the reference specifying unit 232 specifies the reference point P1 relatively arranged at the upper left of the reference points P1 to P4 as the first reference point when the reference specifying image is viewed from the front, and is relative to each other. The reference point P2 arranged at the lower left is specified as the second reference point, the reference point P3 arranged at the lower right is specified as the third reference point, and the reference point is relatively arranged at the upper right. P4 is specified as the fourth reference point. That is, in a quadrangle having reference points P1 to P4 as angles, the first reference point (reference point P1) and the third reference point (reference point P3) are arranged diagonally, and the second reference point (reference point P2) is arranged diagonally. ) And the fourth reference point (reference point P4) are arranged diagonally. The relative positional relationship between the reference points P1 to P4 is associated with the coordinates, the specific image pickup device, and the reference specific image, and is stored in the storage unit 22 (S108).

On the other hand, when the reference points P1 to P4 are specified in the other captured images captured by the specific imaging device (“Y” in S103), the present device 2 ends the reference point specifying process (S100).

Here, in the present embodiment, each image pickup device 1 (1 to n) is fixed in the store. Therefore, the angle of view of each image pickup apparatus 1 (1 to n) is fixed. Therefore, if the reference points P1 to P4 are specified in other captured images captured by the specific imaging device, the positions of the reference points P1 to P4 in the captured images subsequently captured by the specific imaging device do not change. Therefore, when the value is "Y" in the process (S103), the apparatus 2 can execute the correction process (S200) described later using the information of the reference points P1 to P4 acquired in the past.

● Correction processing Next, the correction processing (S200) will be described with reference to FIG.

In the "correction processing (S200)", a conversion pattern used for fluoroscopic conversion of the subject (product shelf, product) captured in the captured image is selected for each captured image, and parameters corresponding to the conversion pattern are calculated. This is a process of performing fluoroscopic conversion of a captured image based on a conversion pattern and parameters corresponding to the captured image.

FIG. 9 is a partial flowchart of the correction process (S200).
FIG. 10 is a flowchart of another part of the correction process (S200).

First, the image acquisition unit 231 acquires one captured image as a processed image from the storage unit 22 (S201).

Next, the image acquisition unit 231 identifies the image pickup device 1 that has captured the processed image based on the position information associated with the processed image (S202).

Next, the pattern selection unit 233 reads out the four reference points P1 to P4 associated with the image pickup device 1 specified by the image acquisition unit 231 and stored in the storage unit 22, and their coordinates (S203).

Next, the pattern selection unit 233 compares the coordinates of the reference points P1 to P4, and among the X coordinate values and the Y coordinate values of the reference points P1 to P4, the maximum value of the X coordinate value and the X coordinate value are The minimum value, the maximum value of the Y coordinate value, and the minimum value of the Y coordinate value are specified (S204).

Next, the pattern selection unit 233 determines the presence or absence of a specific reference point in the X coordinate value and the Y coordinate value of each of the reference points P1 to P4 (S205).

The "specific reference point" is a reference point having coordinates where the X coordinate value is the maximum value or the minimum value and the Y coordinate value is the maximum value or the minimum value.

When the pattern selection unit 233 determines that there is a specific reference point (“Y” in S205), the pattern selection unit 233 determines which of the reference points P1 to P4 is the specific reference point (S206). ..

When the reference point P1 and / or the reference point P3 is a specific reference point (“P1, P3” in S206), the pattern selection unit 233 is a processed image from a plurality of conversion patterns stored in the storage unit 22. The first diagonal conversion pattern is selected as the conversion pattern for perspective conversion of the subject (S207). The details of the first diagonal conversion pattern will be described later.

FIG. 11 is a schematic diagram illustrating a first diagonal conversion pattern.
Each of (a), (b), (c), (d), and (e) in the figure shows an aspect in which fluoroscopic conversion can be performed by the first diagonal conversion pattern. In the figure, a quadrangle formed by four reference points P1 to P4 (hereinafter referred to as “rectangle before correction”) is shown by a broken line, and a point where each reference point P1 to P4 is located after perspective transformation (hereinafter referred to as “correction”). A quadrangle formed by P11, P12, P13, and P14 (hereinafter referred to as “corrected quadrangle”) is shown by a solid line. The alternate long and short dash line in the figure indicates a diagonal line (hereinafter referred to as “first diagonal line”) formed by reference points P1 and P3 (first reference point, third reference point) including at least one specific reference point.

The figure shows that in the first diagonal conversion pattern, the first diagonal line is the reference axis for the perspective transformation. In the figure, the reference points P1 and P3 (first reference point and third reference point) do not move before and after the perspective conversion (the positions of the reference points P1 and P3 and the positions of the correction points P11 and P13 are the same), and the reference points are shown. It is shown that the point P2 (second reference point) moves to the correction point P12, and the reference point P4 (fourth reference point) moves to the correction point P14. That is, the X coordinate value of the correction point P12 is the X coordinate value of the reference point P1 (first reference point), and the Y coordinate value of the correction point P12 is the Y coordinate value of the reference point P3 (third reference point). The X coordinate value of the correction point P14 is the X coordinate value of the reference point P3 (third reference point), and the Y coordinate value of the correction point P14 is the Y coordinate value of the reference point P1 (first reference point). That is, the X coordinate value of the correction points P12 and P14 after the movement of the moving reference points P2 and P4 is the X coordinate value of any of the non-moving reference points P1 and P3, and the Y coordinate value of the correction points P12 and P14 is the reference. It is a Y coordinate value of any of points P1 and P3.

As described above, the "first diagonal conversion pattern" includes the specific reference points (reference points P1 and P3) among the four reference points P1 to P4 and the reference points P1 and P3 located diagonally to the specific reference points. This is a conversion pattern for moving only two reference points P2 and P4 (second reference point and fourth reference point) excluding. That is, the first diagonal conversion pattern is one set of two sets of reference points (reference points P1, P3 and reference points P2, P4) connected by the two diagonal lines of the uncorrected quadrangle, which does not include the specific reference point. This is a conversion pattern in which only the reference points (reference points P2 and P4) of the above are moved to the positions (correction points P12 and P14) of the corrected shape (corrected quadrangle) of the subject. In other words, the first diagonal conversion pattern is a conversion pattern for fluoroscopic conversion of an captured image with the first diagonal as a reference axis. The first diagonal conversion pattern is an example of the first conversion pattern. That is, the first conversion pattern includes the first diagonal conversion pattern. The first diagonal conversion pattern is, for example, stored in advance in the storage unit 22.

Return to FIG.
On the other hand, when the reference point P2 and / or the reference point P4 is a specific reference point (“P2, P4” in S206), the pattern selection unit 233 is selected from a plurality of conversion patterns stored in the storage unit 22. The second diagonal conversion pattern is selected as the conversion pattern for perspective conversion of the subject of the processed image (S208).

FIG. 12 is a schematic diagram illustrating a second diagonal conversion pattern.
Each of (a), (b), (c), (d), and (e) in the figure shows an aspect in which fluoroscopic conversion can be performed by the second diagonal conversion pattern. In the figure, the quadrangle before correction is shown by a broken line, and the quadrangle after correction is shown by a solid line. The alternate long and short dash line in the figure indicates a diagonal line (hereinafter referred to as “second diagonal line”) formed by reference points P2 and P4 (second reference point, fourth reference point) including at least one specific reference point.

The figure shows that in the second diagonal conversion pattern, the second diagonal line serves as the reference axis for the perspective transformation. In the figure, the reference points P2 and P4 (second reference point and fourth reference point) do not move before and after the perspective conversion (the positions of the reference points P2 and P4 and the positions of the correction points P12 and P14 are the same), and the reference points are shown. It is shown that the point P1 (first reference point) moves to the correction point P11, and the reference point P3 (third reference point) moves to the correction point P13. That is, the X coordinate value of the correction point P11 is the X coordinate value of the reference point P2 (second reference point), and the Y coordinate value of the correction point P11 is the Y coordinate value of the reference point P4 (fourth reference point). The X coordinate value of the correction point P13 is the X coordinate value of the reference point P4 (fourth reference point), and the Y coordinate value of the correction point P13 is the Y coordinate value of the reference point P2 (second reference point). That is, the X coordinate value of the correction points P11 and P13 after the movement of the moving reference points P1 and P3 is the X coordinate value of any of the non-moving reference points P2 and P4, and the Y coordinate value of the correction points P11 and P13 is the reference. It is a Y coordinate value of any of points P2 and P4.

As described above, the "second diagonal conversion pattern" includes the specific reference points (reference points P2 and P4) among the four reference points P1 to P4 and the reference points P2 and P4 located diagonally to the specific reference points. This is a conversion pattern for moving only two reference points P1 and P3 (first reference point and third reference point) excluding. That is, the second diagonal conversion pattern is one set of two sets of reference points (reference points P1, P3 and reference points P2, P4) connected by the two diagonal lines of the uncorrected quadrangle, which does not include the specific reference point. This is a conversion pattern in which only the reference points (reference points P1 and P3) of the above are moved to the positions (correction points P11 and P13) of the corrected shape (corrected quadrangle) of the subject. In other words, the second diagonal conversion pattern is a conversion pattern that performs fluoroscopic conversion of an captured image with the second diagonal as a reference axis. The second diagonal conversion pattern is an example of the first conversion pattern. That is, the first conversion pattern includes the second diagonal conversion pattern. The second diagonal conversion pattern is, for example, stored in advance in the storage unit 22.

Return to FIGS. 9 and 10.
When the pattern selection unit 233 determines that there is no specific reference point (“N” in S205), the pattern selection unit 233 has two reference points P1 to P4 (first reference point and fourth reference point) that are not located diagonally. The Y coordinate values of the points) are compared (S209).

When the Y coordinate value of the reference point P1 is smaller than the Y coordinate value of the reference point P4 (“P1 <P4” in S209), the pattern selection unit 233 uses the Y coordinate value and the reference of the reference point P2 (second reference point). Compare with the Y coordinate value of the point P4 (fourth reference point) (S210).

When the Y coordinate value of the reference point P2 is larger than the Y coordinate value of the reference point P4 (“P2> P4” in S210), the pattern selection unit 233 determines that the rotation state of the square before correction is the first rotation state. After making a determination, the first rotation conversion pattern is selected as the conversion pattern for fluoroscopic conversion of the subject of the processed image from the plurality of conversion patterns stored in the storage unit 22 (S211).

The "first rotation state" is a state in which the quadrangle before correction is rotated clockwise with respect to the processed image in the front view of the processed image.

FIG. 13 is a schematic diagram illustrating the first rotation conversion pattern.
In the figure, the quadrangle before correction is shown by a broken line, and the quadrangle after correction is shown by a solid line. Further, the figure shows that the reference point P1 (first reference point) does not move and the reference point P2 (second reference point) moves to the correction point P12 before and after the perspective transformation. In this case, the reference point P1 is the immovable reference point in the present invention. Further, in the figure, the corrected quadrangle is a rectangle (circumscribed rectangle) circumscribing the immovable reference point (reference point P1), the reference point P3 located diagonally to the immovable reference point, and the reference point P4. Indicates that there is. Furthermore, in the figure, among the sides of the corrected quadrangle, the reference point P3 is located on the side connecting the correction points P12 and P13, and the reference point P4 is located on the side connecting the correction points P13 and P14. Is shown. That is, the X coordinate value of the correction points P11 and P12 is the X coordinate value of the reference point P1 (first reference point), and the Y coordinate value of the correction points P12 and P13 is the Y coordinate value of the reference point P3 (third reference point). It is a value, the X coordinate value of the correction points P13 and P14 is the X coordinate value of the reference point P4 (fourth reference point), and the Y coordinate value of the correction points P11 and P14 is the reference point P1 (first reference point). It is a Y coordinate value. That is, the X coordinate value of the correction points P11 to P14 is the X coordinate value of any of the reference points P1, P3, and P4 in contact with the tangent rectangle, and the Y coordinate value of the correction points P11 to P14 is the reference point P1 in contact with the tangent rectangle. It is a Y coordinate value of any of P3 and P4.

As described above, the "first rotation conversion pattern" is a conversion pattern that deforms the uncorrected quadrangle while rotating it counterclockwise with reference to the reference point P1 (first reference point) in the front view. This is a conversion pattern for moving the two reference points P2 to P4 (second to fourth reference points) to the positions (correction points P12 to P14) of the corrected shape (corrected quadrangle) of the subject. The first rotation conversion pattern is a conversion pattern corresponding to the first rotation state, and is an example of the second conversion pattern in the present invention. The first rotation conversion pattern is stored in, for example, in the storage unit 22 in advance.

Return to FIGS. 9 and 10.
Further, when the Y coordinate value of the reference point P2 is equal to or less than the Y coordinate value of the reference point P4 (“P2 ≦ P4” in S210), the pattern selection unit 233 cannot determine the rotational state of the quadrangle before correction (No. 1). It is determined that it is neither the 1st rotation state nor the 2nd rotation state), and the conversion pattern is not selected (S212). At this time, the correction process (S200) ends.

On the other hand, when the Y coordinate value of the reference point P1 is larger than the Y coordinate value of the reference point P4 (“P1> P4” in S209), the pattern selection unit 233 uses the Y coordinate value of the reference point P1 (first reference point). And the Y coordinate value of the reference point P3 (third reference point) are compared (S213).

When the Y coordinate value of the reference point P1 is smaller than the Y coordinate value of the reference point P3 (“P1 <P3” in S213), the pattern selection unit 233 determines that the rotation state of the square before correction is the second rotation state. After making a determination, the second rotation conversion pattern is selected as the conversion pattern for perspective conversion of the subject of the processed image from the plurality of conversion patterns stored in the storage unit 22 (S214).

The "second rotation state" is a state in which the quadrangle before correction is rotated counterclockwise with respect to the processed image in the front view of the processed image.

FIG. 14 is a schematic diagram illustrating a second rotation conversion pattern.
In the figure, the quadrangle before correction is shown by a broken line, and the quadrangle after correction is shown by a solid line. Further, the figure shows that the reference point P4 (fourth reference point) does not move and the reference point P3 (third reference point) moves to the correction point P13 before and after the perspective transformation. In this case, the reference point P4 is the immovable reference point in the present invention. Further, the figure shows that the corrected quadrangle is a circumscribed rectangle circumscribing the immovable reference point (reference point P4), the reference point P2 located diagonally to the immovable reference point, and the reference point P1. show. Furthermore, in the figure, among the sides of the corrected quadrangle, the reference point P1 is located on the side connecting the correction points P11 and P12, and the reference point P2 is located on the side connecting the correction points P12 and P13. Is shown. That is, the X coordinate value of the correction points P11 and P12 is the X coordinate value of the reference point P1 (first reference point), and the Y coordinate value of the correction points P12 and P13 is the Y coordinate value of the reference point P2 (second reference point). It is a value, the X coordinate value of the correction points P13 and P14 is the X coordinate value of the reference point P4 (fourth reference point), and the Y coordinate value of the correction points P11 and P14 is the reference point P4 (fourth reference point). It is a Y coordinate value. That is, the X coordinate value of the correction points P11 to P14 is the X coordinate value of any of the reference points P1, P2, and P4 in contact with the tangent rectangle, and the Y coordinate value of the correction points P11 to P14 is the reference point P1 in contact with the tangent rectangle. It is a Y coordinate value of either P2 or P4.

As described above, the "second rotation conversion pattern" is a conversion pattern that deforms the uncorrected quadrangle while rotating it clockwise with reference to the reference point P4 (fourth reference point) in the front view. This is a conversion pattern for moving the reference points P1 to P3 (first to third reference points) to the position (correction points P11 to P13) of the corrected shape (corrected quadrangle) of the subject. The second rotation conversion pattern is a conversion pattern corresponding to the second rotation state, and is an example of the second conversion pattern in the present invention. The second rotation conversion pattern is stored in the storage unit 22 in advance, for example.

Return to FIGS. 9 and 10.
Further, when the Y coordinate value of the reference point P1 is equal to or higher than the Y coordinate value of the reference point P3 (“P1 ≧ P3” in S213), the pattern selection unit 233 determines that the rotation state of the quadrangle before correction cannot be determined. Then, the conversion pattern is not selected (S212). At this time, the correction process (S200) ends.

Next, the calculation unit 234 calculates the fluoroscopic conversion parameters corresponding to the conversion patterns selected by the pattern selection unit 233 based on the coordinates of the reference points P1 to P4 and the coordinates of the correction points P11 to P14. (S215). In other words, the calculation unit 234 calculates the parameters based on the coordinates (X coordinate value, Y coordinate value) of the reference points P1 to P4. The calculated parameter is associated with the corresponding conversion pattern and stored in the storage unit 22 (S216).

The "parameter" is a value calculated to correct the shape of the entire subject in the processed image so that the quadrangle before correction is fluoroscopically transformed into the quadrangle after correction. That is, for example, the parameter is a coordinate transformation matrix or an argument calculated (generated) for each transformation pattern (for each imaging device 1). The parameters are calculated corresponding to each conversion pattern (captured image captured by each image pickup device 1). That is, the parameters are the parameter (first parameter) corresponding to the first conversion pattern (first diagonal conversion pattern, second diagonal conversion pattern) and the second conversion pattern (first rotation conversion pattern, second rotation conversion pattern). A parameter (second parameter) corresponding to the above is included.

Next, the correction unit 235 reads out the conversion pattern corresponding to the processed image and the parameter corresponding to the conversion pattern from the storage unit 22 (S217). Next, the correction unit 235 corrects the shape of the subject captured in the processed image by a known fluoroscopic conversion process based on the conversion pattern and the parameters corresponding to the conversion pattern (S218). For the known fluoroscopy conversion process, for example, a known API (Application Programming Interface) such as "getPerspectiveTransform" or "warpPerspective" is used. The corrected processed image (hereinafter referred to as “corrected image”) is stored in the storage unit 22 (S219).

FIG. 15 is a schematic diagram showing an example of a corrected image.
In the figure, the corrected quadrangle formed by the correction points P11 to P14 is shown by a broken line. The figure shows that the corrected quadrangle is rectangular.

As described above, the present device 2 identifies the four reference points P1 to P4 of the subject from the captured image by executing the present method, and obtains a conversion pattern corresponding to the captured image based on the respective reference points P1 to P4. Select and correct the shape of the subject captured in the captured image based on the conversion pattern. Therefore, unlike the conventional image processing, the present device 2 does not require estimation of the position / orientation of the reference medium or the image pickup device 1, and can perform fluoroscopic conversion of the subject captured in the captured image based on the captured image. ..

Further, the present device 2 selects the first conversion pattern when there is a specific reference point. As described above, in the first conversion pattern, the subject is fluoroscopically converted with reference to the diagonal line connecting the specific reference point (for example, the reference point P1) and the reference point located diagonally to the specific reference point (for example, the reference point P3). It is a conversion pattern to be performed. Here, for example, in the captured image, the upper portion of the subject captured in a state of looking down from diagonally above is expanded in the X direction and the lower portion is compressed in the X direction (it becomes an inverted trapezoidal shape). In this case, when the subject is perspective-converted with reference to the upper part of the subject, the corrected image becomes an image stretched in the X direction (an image in which the width of the subject is widened), and the subject is perspective-converted with reference to the lower part of the subject. Then, the corrected image becomes an image compressed in the X direction (an image stretched in the Y direction: an image in which the width of the subject is narrowed). However, when the subject is perspective-transformed with respect to the diagonal line as in this method, the widths of the upper part and the lower part of the subject are corrected to the width in the middle of the Y direction of the subject. In other words, when the subject is perspective-transformed with respect to the diagonal line, the aspect ratio of the subject in the corrected image is close to the aspect ratio of the actual object.

On the other hand, the present device 2 selects the second conversion pattern when there is no specific reference point. As described above, the second conversion pattern is a conversion pattern that deforms the quadrangle before correction while rotating it with respect to one immovable reference point (reference point P1, reference point P4) in front view. In the present embodiment, the immovable reference point is determined based on the magnitude relationship of the Y coordinate values of the reference points P1 to P4 that are not diagonally located among the reference points P1 to P4. Here, when there is no specific reference point, the subject is imaged in a rotated state without being significantly deformed. That is, in the captured image, the widths of the upper part and the lower part of the subject are close to the aspect ratio of the actual object. Therefore, by rotating the subject and deforming it as a whole as in the present method, the present device 2 can perform fluoroscopic conversion of the subject captured in the captured image without significantly impairing the aspect ratio. As a result, the aspect ratio of the subject in the corrected image is easy to approximate to the aspect ratio of the actual object.

● Summary According to the embodiment described above, the present device 2 has a reference specifying unit 232 that specifies the coordinates of the four reference points P1 to P4 and the reference points P1 to P4 from the subject captured in the captured image. , A storage unit 22 for storing a plurality of conversion patterns for fluoroscopic conversion of a subject, a pattern selection unit 233 for selecting a conversion pattern for fluoroscopic conversion of a subject from a plurality of conversion patterns based on coordinates, and a pattern selection unit 233 based on the conversion pattern. It includes a correction unit 235 that corrects the shape of the subject captured in the captured image. According to this configuration, the present device 2 does not require estimation of the position / orientation of the reference medium or the image pickup device 1 as in the conventional image processing, and selects a conversion pattern based on the captured image and captures the captured image. The subject captured in the image can be transparently converted.

Further, according to the embodiment described above, the conversion pattern includes the first conversion pattern and the second conversion pattern. In the first conversion pattern, of the four reference points P1 to P4, only the two reference points P1 and P3 (reference points P2 and P4) that are diagonal in the uncorrected quadrangle are the shapes of the subject after correction (after correction). This is a conversion pattern for moving to the position (correction points P11, P13 (correction points P12, P14)) of the quadrangle). The second conversion pattern is a conversion pattern in which only the three reference points P1 to P3 (reference points P2 to P4) are moved to the positions of the corrected shape (correction points P11 to P13 (correction points P12 to P14)). According to this configuration, the present device 2 selects a conversion pattern corresponding to the same positional relationship from a plurality of conversion patterns classified based on the positional relationship (coordinates) of the four reference points P1 to P4, and the same. A corrected image can be generated only by performing fluoroscopic conversion of the captured image based on the conversion pattern. That is, the present device 2 can select a conversion pattern based on the captured image and perform fluoroscopic conversion of the subject captured in the captured image.

Further, according to the embodiment described above, the pattern selection unit 233 has the maximum or minimum X coordinate value among the X coordinate value and the Y coordinate value of each of the reference points P1 to P4, and the Y coordinate. The presence or absence of a specific reference point having coordinates of the maximum value or the minimum value is determined, the first conversion pattern is selected when it is determined that there is a specific reference point, and the second conversion is determined when it is determined that there is no specific reference point. Select a pattern. According to this configuration, the present device 2 can select a conversion pattern based on a simple process of determining the presence or absence of a specific reference point in a captured image.

Furthermore, according to the embodiment described above, the first conversion pattern is located diagonally between the specific reference point (for example, the reference point P1) among the four reference points P1 to P4. Only the reference point (for example, the reference point P3) and the two reference points (for example, the reference points P2 and P4) excluding the reference point are moved to the corrected shape of the subject. According to this configuration, the present device 2 identifies two reference points P1, P3 / P2, P4 to be moved based on the position of the specific reference point, and selects the first diagonal conversion pattern or the second diagonal conversion pattern. can. When the subject is perspective-transformed with respect to the diagonal line, the aspect ratio of the subject in the corrected image approximates the actual aspect ratio. That is, the present device 2 performs fluoroscopic conversion of the subject captured in the captured image so as to approximate the aspect ratio of the actual object by selecting the first diagonal conversion pattern or the second diagonal conversion pattern based on the captured image. can.

Furthermore, according to the embodiment described above, the second conversion pattern includes the immovable reference point (for example, the reference point P1) and the reference point located diagonally to the immovable reference point (for example, the reference point P3). , Three reference points (reference points P2 to P4) excluding the immovable reference point so that the circumscribed rectangle circumscribing to one of the other two reference points (for example, the reference point P4) becomes a corrected quadrangle. To move. According to this configuration, the present device 2 can specify the corrected quadrangle in the second conversion pattern based on the captured image. Then, the second conversion pattern transforms the uncorrected quadrangle into the circumscribed rectangle of the three reference points including the immovable reference point without moving the immovable reference point. As a result, the present device 2 can perform fluoroscopic conversion of the subject captured in the captured image based on the captured image without significantly impairing the aspect ratio even when there is no specific reference point.

Furthermore, according to the embodiment described above, the immovable reference points are related to the magnitude of the Y coordinate values of the two reference points (reference points P1 to P4 in the present embodiment) that are not located diagonally to each other. It is decided based on. According to this configuration, the present device 2 can determine an immovable reference point based on the captured image, select a conversion pattern based on the immovable reference point, and perform fluoroscopic conversion of the subject captured in the captured image.

Furthermore, according to the embodiment described above, when the pattern selection unit 233 determines that there is no specific reference point, the pattern selection unit 233 determines the rotational state of the quadrangle before correction based on the coordinates of each of the reference points P1 to P4. do. The pattern selection unit 233 selects the first rotation conversion pattern when the rotation state is determined to be the first rotation state, selects the second rotation conversion pattern when the rotation state is determined to be the second rotation state, and determines the rotation state. If not, do not select a conversion pattern. According to this configuration, the present device 2 does not execute the perspective conversion when the degree of deformation of the subject in the captured image is large, and executes the perspective conversion based on the captured image when the degree of deformation is small.

In the embodiment described above, the image pickup apparatus 1 and the present apparatus 2 are separate bodies. Instead of this, the image pickup apparatus may function as the present apparatus. That is, for example, the image pickup apparatus may include an acquisition unit, a reference specifying unit, a pattern selection unit, a calculation unit, and a correction unit in the present invention. In this case, the image pickup apparatus may include the display unit and the operation unit according to the present invention, or the external device connected to the image pickup apparatus via a network includes the display unit and the operation unit according to the present invention. May be good.

Further, this device does not have to be provided with a display unit and an operation unit.

Further, the image pickup apparatus in the present invention may be a wide-angle camera capable of capturing a wide range. In this case, the number of image pickup devices arranged in the sales floor of the store is reduced. Therefore, the degree of freedom of wiring for arranging the image pickup device is increased, and the image pickup device can be efficiently arranged in a store where the space where the image pickup device can be installed is small.

Furthermore, the image pickup device in the present invention may be provided in a mobile communication terminal (for example, a smart phone, a tablet, a mobile phone, etc.) carried by a store clerk.

Furthermore, the image pickup apparatus in the present invention and the present apparatus may be installed in a physically separated building.

Furthermore, the image acquisition unit in the present invention may acquire captured images (reference specific image, processed image) directly from each image pickup device via the communication unit in the processing (S101, S201).

Furthermore, the present device may execute the reference point specifying process for each captured image on which the correction process is executed. That is, for example, the present device does not have to execute the determination of the process (S103) in the reference point specifying process.

Further, in the process (S104), the apparatus is configured to display a reference specifying image on the display unit and specify an arbitrary position selected by the user of the apparatus by operating the operation unit as a reference point. You may. In this case, the reference specifying unit in the present invention specifies a pixel selected by the user or a pixel near the selected pixel that can be a reference point (a pixel capable of forming a rectangular area) as a reference point. You may. According to this configuration, the present device can specify the position desired by the user as a reference point. In addition, the processing load of image processing such as sign detection can be reduced.

Furthermore, the reference specifying unit in the present invention only needs to be able to individually specify each reference point, and the positions of the first to fourth reference points are not limited to the present embodiment. That is, for example, the reference specifying unit in the present invention specifies a reference point (reference point P4) located at a relatively upper right position as a first reference point in the captured image, and a reference point (reference point P4) located at a relatively lower right position. The reference point P3) is specified as the second reference point, the reference point relatively lower left (reference point P2) is specified as the third reference point, and the reference point relatively upper left (reference point P1) is specified. May be specified as the fourth reference point. That is, for example, in the process (S107), the reference specification specifies two reference points having a large X coordinate value among the reference points, and the reference point having a small Y coordinate value among the reference points is the first reference point. It may be specified as a point, and a reference point having a large Y coordinate value may be specified as a second reference point.

Furthermore, in the process (S107), the method of specifying the first to fourth reference points is not limited to the present embodiment. That is, for example, the reference specifying unit identifies two reference points having a large Y coordinate among the reference points, and specifies a reference point having a small X coordinate as the first reference point among the reference points, and X A reference point having a large coordinate value may be specified as a fourth reference point. Further, for example, the reference specifying unit may specify the first to fourth reference points based on the distance between the pixel value of each reference point and the pixel value of each of the four corners of the captured image.

Furthermore, the arrangement of each reference point is not limited to the present embodiment as long as each correction point is arranged so as to form a rectangle in the corrected image. That is, for example, each reference point may be arranged at both ends of a shelf board of a certain stage and both ends of a shelf board arranged above the shelf board.

Furthermore, the present device may periodically (for example, every month) execute a reference point specifying process for each of the captured images captured by each imaging device.

Furthermore, in the correction process, when two reference points that do not connect diagonal lines like a trapezoid are specific reference points, the pattern selection unit in the present invention selects either the first diagonal line conversion pattern or the second diagonal line conversion pattern. You may. In this case, for example, the pattern selection unit may be preset to select any first conversion pattern (for example, the first diagonal conversion pattern).

Furthermore, the method executed by the present device is not limited to the present embodiment. That is, for example, the present device may select a conversion pattern in the reference point specifying process and calculate the parameters corresponding to each conversion pattern. Normally, if the posture and angle of view of the image pickup device and the subject (product shelf) do not change, the position of the reference point, the conversion pattern selected, and the calculated parameters do not change in each captured image captured by the same captured image. .. Therefore, the present device can simplify the correction process by storing the conversion patterns and parameters corresponding to each image pickup device in the storage unit in advance. That is, the processing load for generating the corrected image is reduced.

Furthermore, in the embodiment described above, the immovable reference point is the reference point P1 to the reference point P1 to P4 (for example, the first reference point and the fourth reference point) having a larger Y coordinate value among the Y coordinate values. It was P4. Instead, the immovable reference point may be a reference point having a larger Y coordinate value among the Y coordinate values of the second reference point and the third reference point, and the X coordinate value of the first reference point and the second reference point. A reference point having a small X coordinate value may be used, or a reference point having a large X coordinate value among the X coordinate values of the third reference point and the fourth reference point may be used. That is, for example, the immovable reference point may be determined based on the magnitude relationship of the X coordinate values of the two reference points that are not located diagonally.

Furthermore, if the rotation state of the quadrangle before correction can be determined in the process (S209), the coordinate values compared by the pattern selection unit in the present invention are not limited to the Y coordinate values of the first reference point and the fourth reference point. That is, for example, the pattern selection unit in the present invention may compare the Y coordinate values of the second reference point and the third reference point that are not located diagonally. Further, in the process (S209), the pattern selection unit in the present invention may compare the X coordinate values of the first reference point and the second reference point that are not located diagonally. In this case, for example, in the process (S210), the pattern selection unit in the present invention may compare the X coordinate values of the first reference point and the third reference point to determine the same rotation state.

Furthermore, the present device may use the generated corrected image as AI teacher data for image recognition used for extracting product shortages, creating shelving allocations, and the like.

Furthermore, the scene where the system S (device 1) is applied is not limited to the image correction of the retail store. That is, for example, this device can be applied to image correction of aerial photographs, image correction of surveillance cameras in the city, and image correction of landscape photographs.

Furthermore, according to the embodiment described above, the present device 2 is composed of one computer. Instead of this, the present device may be composed of a plurality of computers. That is, for example, this system may be composed of a plurality of computers and may include a group of computers that function as the present apparatus. Specifically, for example, the present apparatus (computer group) may be composed of a computer including a storage unit for storing captured images and a computer including a control unit for executing the present method. Further, for example, a plurality of computers may have the functions of the image acquisition unit, the reference specifying unit, the pattern selection unit, the calculation unit, and the correction unit in a distributed manner. In this case, the plurality of computers may send and receive information via a network, or may transfer information via a portable storage medium.

S Image processing system 1 Image processing device 2 Image processing device 22 Storage unit 23 Control unit 231 Image acquisition unit 232 Reference identification unit 233 Pattern selection unit 235 Correction unit

Claims (16)

1. An image processing device that performs perspective transformation of an image captured in a state in which a subject having a rectangular feature is perspectively projected, and corrects the shape of the subject captured in the captured image. ,
   An image acquisition unit that acquires the captured image, and
   A reference specifying unit for specifying the four reference points forming a quadrilateral region from the subject captured in the captured image and the coordinates of each of the four reference points.
   A storage unit that stores a plurality of conversion patterns for fluoroscopic conversion of the subject, and
   A pattern selection unit that selects the conversion pattern for fluoroscopic conversion of the subject from among the plurality of conversion patterns stored in the storage unit based on the coordinates.
   A correction unit that corrects the shape of the subject captured in the captured image based on the conversion pattern, and a correction unit.
   Have
   An image processing device characterized by this.
2. The conversion pattern is
   A first conversion pattern that moves only two of the four reference points to the position of the corrected shape of the subject, and
   A second conversion pattern that moves only three of the four reference points to the corrected shape position of the subject, and
   including,
   The image processing apparatus according to claim 1.
3. In an array of a plurality of pixels constituting the captured image, when the column direction is the X direction and the row direction is the Y direction.
   The coordinates are
   The X coordinate value indicating the coordinate value in the X direction and the X coordinate value
   The Y coordinate value indicating the coordinate value in the Y direction and the Y coordinate value
   Including
   The pattern selection unit is
   Of the X coordinate value and the Y coordinate value of each of the reference points, the specific reference point having the coordinates of the maximum value or the minimum value of the X coordinate value and the maximum value or the minimum value of the Y coordinate value. Determine the presence or absence,
   When it is determined that the specific reference point exists, the first conversion pattern is selected, and the specific reference point is selected.
   When it is determined that there is no specific reference point, the second conversion pattern is selected.
   The image processing apparatus according to claim 2.
4. In the first conversion pattern, of the four reference points, only two reference points excluding the specific reference point and the reference point located diagonally to the specific reference point are corrected for the subject. Move to the position of the shape of
   The image processing apparatus according to claim 3.
5. In the second conversion pattern, the immovable reference point that does not move among the four reference points and the circumscribed rectangle that circumscribes any two of the other three reference points are the corrected shapes of the subject. The three reference points other than the immovable reference point are moved so as to be.
   The image processing apparatus according to claim 3.
6. The circumscribed rectangle circumscribes the immovable reference point, the reference point located diagonally to the immovable reference point, and any of the other two reference points.
   The image processing apparatus according to claim 5.
7. The immovable reference point is determined based on the coordinates of each of the four reference points.
   The image processing apparatus according to claim 5.
8. The immovable reference point is determined based on the magnitude relationship between the X coordinate value or the Y coordinate value of two of the four reference points that are not located diagonally.
   The image processing apparatus according to claim 7.
9. The pattern selection unit is
   When it is determined that there is no specific reference point, the rotational state in the captured image of the quadrangle is determined based on the coordinates of each of the reference points.
   The rotational state is
   First rotation state and
   Second rotation state and
   Including
   The second conversion pattern is
   The first rotation conversion pattern corresponding to the first rotation state and
   The second rotation conversion pattern corresponding to the second rotation state and
   including,
   The image processing apparatus according to claim 3.
10. The pattern selection unit does not select the conversion pattern when the rotation state cannot be determined.
    The image processing apparatus according to claim 9.
11. The reference specifying unit identifies four reference points based on the sign attached to the subject.
    The image processing apparatus according to claim 1.
12. A display unit that displays the captured image and
    An operation unit that can select an arbitrary position of the captured image displayed on the display unit, and an operation unit that can select an arbitrary position.
    Have
    The reference specifying unit specifies a position selected by operating the operation unit as the reference point.
    The image processing apparatus according to claim 1.
13. To make the computer function as the image processing apparatus according to claim 1.
    An image processing program characterized by this.
14. It is an image processing system that corrects the shape of the subject captured in the captured image by performing perspective transformation of the subject with respect to the captured image captured in a state in which a subject having a rectangular feature is perspectively projected. ,
    An image pickup unit that captures the captured image and
    An image processing device that corrects the shape of the subject captured in the captured image, and
    Have
    The image processing device is the image processing device according to claim 1.
    An image processing system characterized by that.
15. It is executed by an image processing device that performs perspective conversion of the subject to be captured in a state in which a subject having a rectangular feature is perspectively projected and corrects the shape of the subject captured in the captured image. Image processing method
    The conversion pattern in which the image processing apparatus performs perspective conversion of the subject from among a plurality of conversion patterns of the perspective conversion based on the coordinates of four reference points identified from the subject captured in the captured image. Steps to select and
    A step in which the image processing apparatus corrects the shape of the subject captured in the captured image based on the conversion pattern.
    Have
    An image processing method characterized by that.
16. The step in which the image processing device acquires the captured image,
    A step in which the image processing device specifies four reference points forming a quadrilateral region from the subject captured in the captured image and the coordinates of each of the four reference points.
    Have
    The image processing method according to claim 15.

Images