JP2019113976A - Movement line analysis system, control method therefor, and program - Google Patents

Abstract

To provide a system capable of detecting a movement line of a product taken out from a sales section and presenting the movement line to a host store.SOLUTION: A movement line analysis system comprising image capturing devices interconnected with a server includes: movement line information generation means configured to generate movement line information associated with a movement trajectory of a product taken out by a customer using images captured by the image capturing devices; and display control means configured to display location information representing a location at which the product was left on the basis of association between the product and a sales section corresponding to an endpoint of the movement line information generated by the movement information generation means.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a flow line analysis system, a flow line analysis system control method, and a program using a photographed image photographed by a photographing device.

  There is known a system that analyzes flow line information of a person moving in a target area at a shooting site where a camera is installed.

  Patent Document 1 discloses a flow line analysis system in which a camera device installed in a store and a server device are connected to each other, and the flow line analysis in which flow line information of a moving object corresponding to a specific condition is superimposed. A flow line surveillance system for generating an image is disclosed.

JP, 2017-123024, A

  By the way, in the above-mentioned store, a customer may not buy the goods taken out from the counter as a result of examination. In such a case, the customer may return the item to a different place than the counter from which the item was taken out. Therefore, on the shop side, there is a risk that the quality of the product may deteriorate, and the shop may be cluttered. Therefore, it is preferable for the store side to be able to grasp such a situation as quickly as possible and to cope with it. However, in the conventional flow line monitoring system, although it is possible to obtain flow line information of a moving object, these problems have not been dealt with at all.

  Then, this invention is made in view of the said subject, and detects the flow line of the goods taken out from the shop, and it aims at providing the system which can be shown to a store side.

  Therefore, the present invention is a flow line analysis system in which a photographing device and a server are connected to each other, and a movement related to a locus along which a commodity brought out to a customer moves based on a photographed image photographed by the photographing device. The position at which the product is placed is determined based on the correspondence between the product and flow line information generation means for generating line information, and a counter corresponding to the end point of the flow line information generated by the flow line information generation means. And display control means for displaying the position information to be indicated.

  According to the present invention, it is possible to provide a system capable of detecting a flow line of a commodity brought out of a sales department and presenting it to a store.

It is a figure showing an example of the information processing system in this embodiment. It is a figure which shows an example of the hardware constitutions of the server in this embodiment, and a client terminal. It is a figure which shows an example of the hardware constitutions of the imaging device in this embodiment. It is a figure showing an example of functional composition of an information processing system in this embodiment. It is a figure which shows an example of a detection process of the moving body from the picked-up image in this embodiment. It is a flowchart which shows an example of the flow line information generation process in this embodiment. It is a figure which shows an example of the database of the flow line detection result of goods in this embodiment. It is a figure which shows an example of the display screen of the client terminal in this embodiment.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific embodiments of the configuration described in the claims.

  An example of the information processing system in this embodiment is shown in FIG. FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system according to the present embodiment. The information processing system is a system for monitoring the inside of a store by a shooting device disposed in the store, and includes a server 101, a client terminal 102, and a shooting device 103. In addition, the information processing system of this embodiment functions as a flow line analysis system which analyzes the flow line of goods in a store.

  In FIG. 1, one or more photographing devices (network cameras) 103 are arranged on the floor in the store. The photographing device 103 is disposed at the entrance of the store, at a position where each counter can be photographed. Further, when there are a plurality of floors (floor A and floor B) in the store, a plurality of imaging devices 103 are arranged in units of each floor. Each imaging device 103 is connected to the server 101 via the network 104.

  The network 104 communicably connects the server 101, the client terminal 102, and the photographing apparatus 103 to each other. The network 104 is composed of, for example, a plurality of routers, switches, cables, and the like that satisfy communication standards such as a local network. In the present embodiment, the communication standard, the scale, and the configuration do not matter as long as the apparatuses can communicate with each other. For example, the network 104 may be configured by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), or the like.

  Next, an example of the hardware configuration applicable to the server 101 and the client terminal 102 will be described using FIG.

  In FIG. 2, reference numeral 201 denotes a CPU, which centrally controls devices and controllers connected to the system bus 204. Further, the ROM 202 or the external memory 211 may be a BIOS (Basic Input / Output System) which is a control program of the CPU 201, an operating system program (hereinafter referred to as an OS), and various kinds of later described necessary for realizing functions executed by the PC. Programs and the like are stored.

  A RAM 203 functions as a main memory, a work area, and the like of the CPU 201. The CPU 201 loads programs necessary for execution of processing from the ROM 202 or the external memory 211 to the RAM 203, and implements various operations by executing the loaded programs.

  An input controller 205 controls an input from a pointing device such as a keyboard (KB) 209 or the like. Reference numeral 206 denotes a video controller which controls display on a display such as a display 210 (whether liquid crystal or CRT).

  A memory controller 207 is an external storage device (hard disk (HD)) for storing a boot program, various applications, font data, user files, editing files, various data, etc., a flexible disk (FD), or a PCMCIA card slot. It controls access to external memory 211 such as Compact Flash (registered trademark) memory connected via an adapter.

  A communication I / F controller 208 connects and communicates with an external device via a network, and executes communication control processing in the network. For example, communication using TCP / IP is possible.

  Note that the CPU 201 enables display on the display 210 by executing, for example, outline font rasterization processing on a display information area in the RAM 203. In addition, the CPU 201 enables user instruction with a mouse cursor (not shown) or the like on the display 210.

  Various programs to be described later for realizing the present embodiment are stored in the external memory 211, and are executed by the CPU 201 by being loaded into the RAM 203 as necessary. Furthermore, setting files and the like used at the time of execution of the program are also stored in the external memory 211, and a detailed description of these will be described later. This is the end of the description of FIG.

  Next, an example of the hardware configuration of the imaging device 103 illustrated in FIG. 1 will be described using FIG. 3. FIG. 3 is a diagram illustrating an example of a hardware configuration of the imaging device 103 in the present embodiment.

  The CPU 301 centrally controls the devices and controllers connected to the system bus 304. Further, the ROM 302 or the external memory 305 may include a BIOS (Basic Input / Output System) that is a control program of the CPU 301, an operating system program (hereinafter referred to as an OS), and a function described later. Various programs etc. are stored. The RAM 303 functions as a main memory, a work area, and the like of the CPU 301.

  The CPU 301 loads various programs and the like necessary for execution of processing into the RAM 303 and executes various programs to realize various operations.

  A memory controller (MC) 306 is a CF connected via an adapter to a hard disk (HD) that stores a boot program, various applications, font data, user files, editing files, various data, image data, etc. and a PCMCIA card slot. It controls access to the external memory 305 such as memory and smart media (registered trademark).

  The camera unit 307 is connected to the image processing unit 308, and photoelectrically converts light obtained by transmitting light through a lens directed to a monitoring target by a light receiving cell such as a CCD or CMOS, and then converts the light into an RGB signal. And a complementary color signal to the image processing unit 308.

  The image processing unit 308 performs white balance adjustment, gamma processing, and sharpness processing based on RGB signals and complementary color signals, and further performs YC signal processing to generate a luminance signal Y and a chroma signal (hereinafter, YC signal). , YC signals in a predetermined compression format (for example, JPEG format or Motion JPEG format), and the compressed data is temporarily stored in the external memory 305 as image data.

  A communication I / F controller (communication I / FC) 309 is for connecting and communicating with an external device via a network, executes communication control processing in the network, and stores the image stored in the external memory 305. The data is transmitted by the communication I / F controller 309 to an external device.

  Next, an example of a functional configuration of the information processing system in the present embodiment will be described using FIG. 4. The functional configuration shown in FIG. 4 is merely an example, and there are various configuration examples according to the application and purpose, and thus the functional configuration is not limited to this. Each of these functions is executed by the CPU 201 of the server 101, the CPU 201 of the client terminal 102, the CPU 301 of the photographing apparatus 103, or the cooperation among them.

  The imaging device 103 includes a captured image acquisition unit 431, a product detection unit 432, and a flow line information generation unit 433.

  The photographed image acquisition unit 431 is a functional unit that acquires a photographed image. The captured image is an image of each frame of the moving image captured by the camera unit 307. The captured image acquisition unit 431 acquires a captured image at a predetermined frame rate (for example, 20 fps).

  The commodity detection unit 432 performs a predetermined image process (for example, a person detection process and an object detection process) on the frame of the captured image output from the captured image acquisition unit 431 so that the moving object included in the frame of the captured image The presence or absence of (for example, a person and an object) is detected. When the commodity detection unit 432 detects a moving object included in a frame of a captured image, information on the moving object with respect to the frame of the captured image and information on a detection region of the moving object (for example, coordinate information of the moving object in the frame) It is output to the flow line information generation unit 433. The commodity detection unit 432 outputs information (for example, predetermined null information) on the detection area of the moving object to the movement line information generation unit 433 when the moving object included in the frame of the captured image is not detected. . Further, the product detection unit 432 specifies whether only the person is moving or the person is moving with a predetermined product.

  Here, processing for separating a moving object from a captured image will be described with reference to FIG.

  Reference numeral 500 denotes a photographed image of three frames (frame 1, frame 2, frame 3) of a moving image photographed by the photographing device. The moving object separation image 510 is an image obtained by extracting only the moving object from the captured image 500. In FIG. 5, the photographed image 500 and the moving object separation image 510 show an example in which time changes from the upper side to the lower side of the paper surface. Here, the case where the person 501 who has visited the store takes out one product 503 out of the plurality of products 502 stacked up will be described. Further, the images of the same frame number out of the photographed image 500 and the moving object separation image 510 are in correspondence with each other. In each frame of the photographed image, the information of the moving body (for example, the person 501 and the product 503) and the information of other than the moving body (for example, background or 504) are separated. In frames 2 and 3 of the captured image showing that the person 501 who is a moving object carries the product 503, the values of luminance and color difference corresponding to the pixels of the product 503 carried by the person 501 change along the time axis direction. . On the other hand, the goods 504 which are not moving objects do not change along the time direction. Therefore, the commodity detection unit 432 can regard the commodity 503 carried by the person 501 as a moving object. Therefore, the product detection unit 432 can specify whether only the person 501 is moving or whether the person is moving with a predetermined product. Further, the product detection unit 432 can specify a moving product and a non-moving product among the plurality of products. In addition, the product detection unit 432 may identify the brought-out product by comparing the product that has not moved and the product that is moving. This is because when the person holds the product in hand, all the features may not appear in the photographed image, and the processing in the image analysis may be difficult. Hereinafter, the description of FIG. 5 returns to the description of FIG. 4.

  The movement line information generation unit 433 is based on the result detected by the product detection unit 432, information on the photographed image output from the photographed image acquisition unit 431 and information on the detection area of the moving object in the past (for example, the photographed image The coordinates) are used to associate information on current and past detection areas. The movement line information generation unit 433 associates information on the captured image with information on the present and past detection areas, and outputs the information to the server 101 as movement line information. That is, the flow line information is the change amount of the coordinate information of the moving body, and can also be said to be information on the trajectory of the commodity brought out by the customer. Also, it can be said that the flow line information generation unit 433 generates flow line information based on a change in a captured image between frames. The current detection area is, for example, the detection area of frame 3 in FIG. 5, and the past detection area is, for example, the detection area of frame 2. When acquiring the flow line information, the flow line information generation unit 433 may associate the product with the customer who brought out the product, and may generate the flow line information of the product using the flow line information of the customer. This is because in the case of a small product, it may be a blind spot in the shooting range. Therefore, the flow line information generation unit 433 can improve the accuracy of flow line detection by detecting the operation carried out by the customer and associating the customer with the product.

  The server 101 includes a flow line information analysis unit 411 and a display control unit 412.

  The movement line information analysis unit 411 is based on the movement line information output from the movement line information generation unit 433 for a plurality of photographed images, and the staying position or passing position of the moving object (person and goods) in the frame of the photographed image Extract and generate flow line information on The movement line information analysis unit 411 extracts, using the movement line information for the frames of the plurality of photographed images, accurate movement line information on the position where the moving object stagnates or the position where it passes through in the frames of the photographed image. Can be generated.

  The movement line information analysis unit 411 can acquire movement line information (a locus of movement of a commodity brought out by a customer) from the plurality of imaging devices 103. The movement line information analysis unit 411 can acquire the locus of goods on the entire floor of the store by combining the movement line information from the plurality of imaging devices 103. Specifically, the movement line information analysis unit 411 combines pieces of movement line information whose imaging times are the same between the imaging devices 103. Furthermore, the movement line information analysis unit 411 combines a plurality of pieces of movement line information based on movement line information in which the imaging ranges of the plurality of imaging devices 103 overlap or are adjacent to each other.

  Furthermore, the movement line information analysis unit 411 specifies the start point and the end point of the movement line from the movement line information of the moving object. For this reason, the flow line information analysis unit 411 can specify to which department the product brought out to the customer has moved from the original department. That is, the flow line information analysis unit 411 sets the position at which the product brought out to the customer is placed based on the correspondence between the store and the product corresponding to the end point of the flow line information generated by the flow line information generation unit 433. Can identify the position information indicating. The movement line information analysis unit 411 may specify the attribute (name) of the item brought out to the customer by image analysis, or can specify it from the coordinates of the start point of the movement line information. Specifically, the flow line information analysis unit 411 can specify the attribute of the item brought out to the customer based on the correspondence between the coordinates of the start point of the flow line information and the coordinates of the sales floor in the floor layout.

  The display control unit 412 controls the client terminal 102 to display the flow line information analyzed by the flow line information analysis unit 411 and the position information indicating the position where the product is placed. The screen displayed by the display control unit 412 will be described later with reference to FIG.

  The client terminal 102 includes an operation unit 421 and a display unit 422.

  The operation unit 421 is a functional unit that receives an input operation from a user. Specifically, it is a functional unit that receives an input operation to the input device 209 or the like. In the client terminal 102, when the input device 209 and the display 210 are integrated as in the touch panel, the display 210 functions as the input device 209.

  The display unit 422 is, for example, the display 210, and displays a display screen under the control of the display control unit 412 of the server 101. FIG. 8 is an aspect of a display screen displayed on the client terminal 102. Note that FIG. 8 will be described later.

  Next, flow line information generation processing according to the present embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing an example of flow line information generation processing in the present embodiment.

  In step S601, the imaging apparatus 103 acquires a captured image. Here, each of the plurality of photographing devices 103 disposed on the floor performs photographing with a moving image.

  In step S602, the imaging apparatus 103 detects a moving object. The imaging apparatus 103 executes the process of step S603 when a moving object (product) other than a person is detected among the detected moving objects.

  In S603, the imaging device 103 generates flow line information based on the detected information of the moving object. As described in FIG. 4, the method of generating flow line information is generated from captured images of a plurality of frames of a moving image.

  In step S604, the imaging apparatus 103 transmits flow line information to the server.

  In step S605, the server 101 combines flow line information from a plurality of imaging devices. The connection process generates flow line information of the entire floor of the identified product.

  In step S606, the server 101 determines whether the end point of the flow line is a POS cash register terminal. Specifically, the server 101 determines whether the coordinates of the POS cashier terminal on the floor are within a predetermined range from the end point of the flow line. The server 101 does not store the information when the end point of the flow line is a POS cash register terminal. This is because there is no need to dare to notify that the product has been properly accounted. The server 101 may not store the information when the product brought out by the customer is returned to the original store, that is, when the start point and the end point of the flow line substantially match. This is to indicate that the product has been properly returned to its original position. Then, when the end point of the flow line is not the POS cash register terminal, the server 101 executes the processing of S 607.

  In S607, the server 101 determines whether the counter corresponding to the end point of the flow line matches the attribute of the product. The server 101 does not store the information when the section corresponding to the end point of the flow line matches the attribute of the product. This is because there is no need to dare to notify that the product has been returned to the appropriate position. On the other hand, when the counter corresponding to the end point of the flow line and the attribute of the product do not match, the server 101 executes the process of S608.

  In S608, the server 101 determines the degree of urgency based on the correlation between the counter corresponding to the end point of the flow line and the attribute of the product.

  In S609, the server 101 stores the above-described flow line information in the external memory 211 of the server 101.

  In S610, the server 101 displays a display screen based on flow line information on the client terminal. The above is the description of the flow line information generation process in the present embodiment.

  Next, a database for storing the result of the flow line information generation process will be described with reference to FIG. FIG. 7 is a diagram showing an example of a database of flow line detection results of goods. The database 700 is a list showing the relationship between the detected item and the counter where the item is placed.

  The date and time 701 is the time when the product 702 is placed in the counter 703. The date and time 701 is not limited to the time placed in the sales department 703, and may be a time taken out or a time taken from taking out to being placed in the sales department.

  An item 702 indicates an item taken out of the department where flow line generation has been generated. That is, the product 702 is a list of products taken out of the counter to the customer, not purchased, and returned to the counter different from the original counter.

  The counter 703 is a counter where the product 702 is returned.

  The degree of urgency 704 is information determined based on the attributes of the product 702 and the counter returned. Specifically, the degree of urgency 704 is set lower as the attributes of the product 702 and the counter returned are closer. On the other hand, the degree of urgency 704 is set higher as the attribute between the product 702 and the counter returned is longer. For example, when the product "ice" is returned to the department "dairy product", it is highly likely that the product has been left at a temperature lower than the normal storage state. Therefore, since the quality of ice may deteriorate, the degree of urgency is "medium". As another example, if the product "sweater" is returned to the department "underwear", the department becomes complicated, but the possibility of quality deterioration is low, so the degree of urgency is "low". Then, when the product "fish" is returned to the sales section "daily commodity", the possibility of deterioration of quality is extremely high if it is left, so the degree of urgency is "high". The above is the description of the database 700.

  Next, a display screen displayed on the client terminal 102 will be described with reference to FIG. FIG. 8 is a view showing an example of a display screen of the client terminal 102. As shown in FIG. The floor layout screen 800 in FIG. 8 is a screen showing the floor layout. The floor layout screen 800 is a layout screen created in advance by CAD or the like. The photographing device 103 in FIG. 8 shows the arrangement of the photographing device 103 in the floor, and is not displayed on the actual floor layout screen 800. The floor layout screen 800 is a display screen in a state in which flow line information is not displayed, and the flow line information display screen 810 is a display screen in a state in which flow line information is displayed. Here, how the data of the database 700 is displayed on the flow line information display screen 810 will be described. The movement line information 811 is movement line information of the product “ice”, and indicates by an arrow that the frozen food department has moved to the dairy department. Similarly, the flow line information 812 is flow line information on the product "sweater". The movement line information 813 is movement line information on the product "fish". The flow line information 831 is a flow line when the item brought out by the customer is carried to the POS cash register terminal and purchased. When displaying the flow line (for example, flow line information 831) of the purchased product, the flow line information display screen 810 can distinguish these from those in which the product is not arranged at an appropriate position (for example, a broken line) And solid lines etc.). The flow line information display screen 810 may not display the flow line information 831.

  Position information 821, position information 822, and position information 823 are displayed at a position corresponding to the end point of the flow line information. The position information 821 is a display indicating the position where the product “ice” is placed. The position information 821 indicates the urgency 704 of the database 700. In the position information 821, "Medium" is displayed as the degree of urgency 704 of the product "Ice". The position information 822 is a display related to the product “sweater”, and the position information 823 is a display related to the product “fish”.

  In addition, the flow line information display screen 810 may change and display a display mode for every attribute of goods, and may display only goods of a specific attribute. These can be switched with each other by the operation of the user. Furthermore, the flow line information display screen 810 may change and display the display mode for each degree of urgency, or may display only goods of a particular degree of urgency, which are mutually switched by the user's operation. it can. Furthermore, the flow line information display screen 810 may display only flow line information of a product for a specific period. The specific period is, for example, one day, six hours, and the like. Then, the flow line information display screen 810 may specify and display the attribute of the product, the degree of urgency, and the period described above. For example, on the flow line information display screen 810, it is possible to set the product "Meat", the degree of urgency "High", the period "3 hours", and the like.

  As mentioned above, according to the above-mentioned this embodiment, an information processing system is a flow line analysis system by which an imaging device and a server were connected mutually. Then, the flow line analysis system generates flow line information on the trajectory of the product taken out by the customer based on the photographed image photographed by the photographing device (flow line information generation unit 433). The position information indicating the position where the product is placed is displayed based on the correspondence between the counter and the product corresponding to the end point (FIG. 7, FIG. 8, etc.). As a result, it is possible to provide a system capable of detecting the flow line of goods brought out of the counter and presenting it to the store.

  Further, the flow line analysis system displays the position information based on the correspondence between the start point and the end point of the generated flow line information. As a result, the flow line analysis system can not display flow line information or position information when the accounting is properly performed or when the product is returned to the original store. For this reason, there is also an effect that the relationship between the product and the position where the product is placed can be intuitively easily grasped simply by looking at the flow line information display screen.

  Furthermore, the flow line analysis system associates a product and a customer who has brought the product on the basis of a captured image captured by the imaging device, and generates flow line information of the product using the flow line information of the customer. There is. This makes it possible to easily acquire the flow line information of the product even when the product is difficult to see from the shooting range.

  Note that the present embodiment can also be realized by a computer or a control computer executing a program (computer program). In addition, means for supplying a program to a computer, for example, a computer-readable recording medium such as a CD-ROM recording the program or a transmission medium such as the Internet transmitting the program can be applied as an embodiment. . The above program can also be applied as an embodiment. The above program, recording medium, transmission medium and program product are included in the scope of the present invention.

  As mentioned above, although it explained in full detail based on an embodiment, it is not limited to a specific embodiment, and various forms of the range which does not deviate from the gist of an invention are also included in the category of the present invention. Furthermore, the above-described embodiment is merely an example, and the invention easily conceivable from the above-described embodiment is also included in the scope of the present invention.

101 server 103 imaging device 411 flow line information analysis unit 412 display control unit 431 captured image acquisition unit 432 product detection unit 433 flow line information generation unit

Claims (8)

A flow line analysis system in which an imaging device and a server are mutually connected,
Flow line information generation means for generating flow line information on a locus along which a commodity brought out by a customer has moved, based on a photographed image photographed by the photographing device;
Display control means for displaying position information indicating a position at which the product is placed, based on the correspondence between the item and the counter corresponding to the end point of the flow line information generated by the flow line information generation means; A flow line analysis system characterized by comprising.   The flow line according to claim 1, wherein the display control means displays the position information based on the correspondence between the start point and the end point of the flow line information generated by the flow line information generation means. Analysis system.   The display control means displays the position information when the end point of the flow line information generated by the flow line information generation means is different from the start point of the flow line information. Flow line analysis system.   The display control means is for when the end point of the flow line information generated by the flow line information generation means is different from the start point of the flow line information, and is different from the position corresponding to the POS register terminal. The flow line analysis system according to any one of claims 1 to 3, wherein the flow line is displayed.   The display control means displays the position information indicating different degrees of urgency based on the end point of the flow line information generated by the flow line information generation means. Flow line analysis system described in.   The flow line information generation unit associates the product and a customer who has brought the product on the basis of a photographed image captured by the photographing device, and uses flow line information of the customer to flow line information of the product. The flow line analysis system according to any one of claims 1 to 5, characterized in that A control method of a flow line analysis system in which an imaging device and a server are mutually connected,
A flow line information generation step of generating flow line information on a movement track of a commodity brought out by a customer based on a photographed image photographed by the photographing device;
A display control step of displaying position information indicating the position at which the product is placed, based on the correspondence between the item and the counter corresponding to the end point of the flow line information generated in the flow line information generating step; A flow line analysis system characterized by comprising.   The program for functioning a computer as each means of the flow line analysis system as described in any one of Claim 1 to 6.

Images