JP2019057849A - Program, information processing apparatus, and image processing system - Google Patents

Abstract

To provide a program for allowing update of an image to be output during imaging.SOLUTION: There is provided a program for causing an information processing apparatus to function as image acquisition means that acquires an image picked up by an imaging device as an input image; subject image determination means that determines whether the input image acquired by the image acquisition means is an image of a subject; determination means that, when the input image is determined to be the image of the subject, determines to output the input image, and when the input image is determined not to be the image of the subject, determines to output an output image output to the outside; and output means that outputs the input image or the output image according to a result of determination made by the determination means.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a program, an information processing apparatus, and an image processing system.

  There is a service in which an imaging device periodically captures the same direction from the same place and provides the current state of the imaging range in real time. In surveillance cameras and consumer behavior surveys, the trend of moving objects such as people is the object of observation, but the background other than moving objects such as persons is often the object of observation.

  As an example of a case where the background is an object of observation, it is considered to distribute an image of a product shelf in an actual store to a terminal device such as a product purchaser who is a user for an EC (Electronic Commerce) site in real time. . Since the user of the EC site can see the actual product shelves with the terminal device, he / she can enjoy shopping with a sense of reality even though it is mail order. However, there are many cases where a customer in front of a product shelf is shown in an image of the inside of an actual store. Providing an image showing a visitor as it is lacks privacy protection, and if an image in which the product is hidden by the visitor is provided to the terminal device, the user cannot view the product.

  Therefore, a technique for removing moving objects other than the background from the captured image is known (for example, see Patent Document 1). In Patent Document 1, a moving object detection unit that detects a frame in which a moving object is displayed is compared with a detected frame in which a moving object is no longer detected and each detection frame in which a moving object before the detected frame is detected. Thus, the detection frame in which the image information is drawn from the detection frame is specified as the update frame, the drawing frame specifying unit for specifying the area of the drawn image information, and the detection frame is replaced with the frame before the detection frame, and the detection is performed. There is disclosed a video editing apparatus including a synthesis processing unit that superimposes a corresponding image information area on an image in a corresponding area of a detected frame when the frame is an update frame in which the image information is drawn.

  However, since the conventional technique is a technique that removes moving objects from a time-series image for a certain period, there is a problem in that an image output while imaging cannot be updated. For example, the conventional technology is a method of performing a process of erasing a moving object and drawing image information from the time when a post-detection frame in which a moving object is no longer detected is detected. Handwritten image information is not displayed.

  An object of this invention is to provide the program which can update the image output, imaging in view of the said subject.

  The present invention relates to an information processing device, an image acquisition unit that acquires an image captured by an imaging device as an input image, and a subject image determination unit that determines whether the input image acquired by the image acquisition unit is an image of a subject. And determining means for determining that the input image is to be output when it is determined that the input image is an image of a subject, and for outputting an output image output to the outside when it is determined that the input image is not an image of a subject. A program for functioning as an output unit that outputs the input image or the output image according to the determination result of the determination unit is provided.

  It is possible to provide a program that can update an output image while imaging.

It is an example of the figure explaining the removal process of the image of the goods shelf imaged in time series, and a moving body. It is an example of the figure explaining how the image imaged periodically changes. It is an example of the figure explaining the time series input image and the output image output from it. 1 is an example of a schematic configuration diagram of an image processing system. It is a figure explaining an example of arrangement of a store image and an imaging device. It is an example of the hardware block diagram of an imaging device. It is an example of the hardware block diagram of the communication terminal in the case of the cradle with a wireless communication function. It is an example of the hardware block diagram of an image management apparatus and a terminal device. It is an example of each functional block diagram of an imaging device, a communication terminal, an image management device, and a terminal device that the image processing system has. It is an example of the figure explaining the magnitude | size of the image after a division | segmentation. It is an example of the figure explaining the determination method of the output image based on similarity. It is an example of the sequence diagram which shows the whole procedure in which an image processing system provides an image. It is a figure which shows an example of the shop list screen which the terminal device displayed. It is a figure which shows an example of the goods screen which the terminal device displayed. It is an example of the flowchart figure which shows the procedure in which an image management apparatus determines an output image. It is an example of the figure explaining the inconvenience which an image processing system solves. FIG. 10 is an example of a diagram illustrating a method for determining an output image based on similarity (second embodiment). FIG. 10 is an example of a diagram illustrating a method for determining an output image based on similarity (second embodiment). FIG. 10 is an example of a flowchart for explaining a procedure for updating an output image by the image processing system (second embodiment); FIG. 10 is an example of a functional block diagram of an imaging device, a communication terminal, an image management device, and a terminal device included in an image processing system (third embodiment). It is an example of the figure explaining an expansion process. It is an example of the figure explaining the suitable example of 8 neighborhood expansion processing. It is an example of the flowchart figure which an expansion process part performs an expansion process with respect to the image after a division | segmentation. FIG. 10 is an example of a functional block diagram of an imaging device, a communication terminal, an image management device, and a terminal device included in an image processing system (Example 4). It is an example of the figure explaining the determination method of the output image using a learning model. It is an example of the flowchart figure which shows the procedure in which a learning model discards an input image. It is an example of the figure explaining the determination method of the output image using a learning model. FIG. 10 is an example of a flowchart illustrating a procedure for determining an output image by an image management apparatus (Example 4).

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Comparative example of image processing for removing moving objects from an image>
In describing this embodiment, a comparative example of moving object removal for removing a moving object from an image will be described.

  FIG. 1 is an example of a diagram illustrating an image of a product shelf 81 captured in time series and a moving object removal process. The imaging device is arranged at a position for imaging at least a part of the product shelf, and periodically images the product shelf. In this way, imaging by the imaging apparatus 1 at the same place may be referred to as fixed point observation. The imaging time interval is a time interval at which an image can be provided early without providing an old image indefinitely when the product shelf changes, and is determined in consideration of the speed with which the product can be sold. Such video distribution is called real-time distribution or live distribution. In determining the time interval, network bandwidth and processing load such as image processing are taken into consideration. The shorter the time interval, the greater the real-time property, but if the product shelf does not change in such a short time, the processing load also increases. From the above, as an example, in this embodiment, the time interval is 1 second. However, the image may be captured at a short time interval such as 30 frames / sec, or may be captured at 2-10 frames / sec. One minute or more may be set as the time interval.

  Fig.1 (a) shows the image of the goods shelf 81 imaged in time series. There is no change in the image 83_1, the image 83_2, the image 83_4, and the image 83_5, but the person 82 is shown in the image 83_3. When the image processing system provides the image 83_3, the person 82 is removed. As a method for removing the person 82, an average image is created by taking an average for each pixel of the image for a predetermined time in the past. . If the staying time of the person 82 is 1 second and the predetermined time is 30 seconds, for example, the influence of the image 83_3 on the average image is 1/30, so an average image in which the person 82 is not shown is obtained.

  However, the person 82 in the store stays for at least a few seconds in front of the same merchandise shelf 81 to color the merchandise rather than simply passing through. For this reason, the translucent person 82 appears in the average image. FIG. 1B schematically shows an average image 84 in which the translucent person 82 is shown. More specifically, the translucent person 82 gradually becomes darker with time as the ratio of images in which the person 82 is captured increases, and the translucent person 82 gradually becomes thinner with time as the ratio decreases. It will become. If the number of images used for the average is increased, the person 82 can be removed, but the change of the product is also reflected in the image later.

  As described above, the removal of moving objects by the average image may be effective if the number of images in which the person 82 is captured is smaller than the number of images that is a parameter of the average image, but the person 82 who looks at the product is captured. It may not be suitable in the store.

<Example of Image Change Focused on by this Embodiment>
Therefore, in the present embodiment, a moving object such as the person 82 is detected by comparing the actually output output image and the captured input image. If a moving object is detected, a past output image in which no moving object is detected is detected. The moving object is removed by the moving object removing method to be output. An output image is an image provided to a user or an image to be displayed, and an input image is a captured image.

  FIG. 2 is an example of a diagram illustrating how an image captured periodically changes, and FIG. 3 is a diagram illustrating a time-series input image and an output image output from the time-series input image. It is an example. FIG. 2 shows a change in an image in which the product shelf 81 on which drinks are placed is periodically imaged. The image is processed in a state of being divided into blocks, and the division into blocks will be described later.

  FIG. 2 shows the degree of similarity between the output image and the input image with respect to the passage of time (takes a value of 0 to 1 and the higher the value, the higher the degree of similarity) 85 and an image example at that time.

Time t ₀ : The product shelf 81 is imaged, and an input image 86 in which the product is reflected (hereinafter, distinguished by attaching time t _{0 to} t _n ) is obtained. The output image is the input image 86_t _0.

Time t ₁ : An input image 86_t ₁ showing a product is obtained, but there is no change with respect to the immediately preceding input image 86_t _{0 and the} similarity is almost 1. The output image is the input image 86_t _1.

Time _{t 2:} Because the person 82 enters the imaging range, the input image 86_T ₂ a person 82 is reflected is obtained. Also, since the similarity is greatly reduced, the output image at time t1 is not updated. This process will be described with reference to FIG.

Time t ₃ : Since the person 82 is continuously imaged, an input image 86_t _{3 in} which the person 82 is captured is obtained. Also, since the similarity is greatly reduced, the output image at time t1 is not updated.

Time t ₄ : Since the person 82 is continuously imaged, an input image 86_t _{4 in} which the person 82 is captured is obtained. Further, since the degree of similarity is greatly reduced, the output image at time t ₁ is not updated.

Time t ₅ : Since the person 82 is continuously imaged even if the part of the person 82 to be imaged changes, an input image 86_t _{5 in} which the person 82 is captured is obtained. Similarity varies but remains lowered, the output image at time t ₁ is not updated.

Time t ₆ : When the person 82 leaves the imaging range, an input image 86_t _{6 in} which the product shelf 81 is captured is captured. Since the degree of similarity is high, the output image is updated to the input image 86_t _6. However, since there is no product at time t ₆ , the similarity is slightly less than 1.

3A shows the similarity 85 with respect to time, FIG. 3B shows the input images 86_t _{0 to} 86_t ₆ , and FIG. 3C shows the output image 87 (hereinafter, times t _{0 to} t _n are attached). To distinguish). Since the image desired to be provided by the image processing system of the present embodiment is the product shelf 81, the input images 86_t _{2 to} 86_t _{4 in} which the person 82 is shown should be removed. The image to be removed is determined from the similarity 85 being less than the threshold.

The image processing system provides a past output image instead of the removed input images 86_t _{2 to} 86_t ₄ . The past output image is, for example, the newest output image (the newest output image 87_t ₁ whose similarity is equal to or greater than a threshold value).

  As described above, the image processing system according to the present embodiment outputs a past output image in which the person 82 is not captured if the person 82 is captured in a certain input image. Therefore, it is possible to provide an image of the commodity shelf 81 as a background by removing an input image in which a moving object such as a person 82 is captured. Since the output image before the moving object is detected is displayed while the moving object is captured, the viewer cannot browse the product. Even if the product on the product shelf 81 is taken out and the image on the product shelf 81 is changed, the decrease in the similarity is slight, so that the output image can be updated with the input image of the product shelf with the reduced number of products.

<Terminology>
An input image refers to an image input to the information processing apparatus regardless of whether it is output. Not only the image received from the imaging device but also an image stored in a storage medium may be used.

  An output image is an image output to the outside, and all or part of the input image becomes an output image. The output destination may be a display device such as a display, an information processing device connected via a network, or a storage medium. The output by the output means includes output to a display device, transmission to an external device, and storage in a storage medium.

  A moving object means a thing provided with the function to move. For example, there are humans, animals, vehicles and bicycles, carts (carts). A moving object is a moving object that moves as a person moves. For example, a hand bag is a moving object, and a drink held by a human is also a moving object. On the other hand, drinks as commodities are not moving because they do not have a function to move and are not held by humans.

  A subject is an object to be copied. What is the subject depends on the purpose of imaging and can be changed. What is between the subject and the imaging device may be referred to as a moving object. The subject is also called a subject to be photographed.

  A background image refers to an image of a product shelf or an image in which no moving object is shown. The subject may be included in a background image such as a product shelf, or the background image may be an image of the subject.

<System configuration example>
FIG. 4 is an example of a schematic configuration diagram of the image processing system 200. The image processing system 200 includes an image management device 5, an imaging device 1, a communication terminal 3, and a terminal device 7 connected via a communication network 9. The imaging device 1 is installed in the store by the installer X. The terminal device 7 is operated by the viewer Y.

  The communication network 9 is constructed to include at least one of the LAN of the company to which the store Y or the viewer Y belongs, the provider network of the provider that connects the LAN to the Internet, and the line provided by the line operator. Yes. When the communication terminal 3 or the terminal device 7 is directly connected to the line telephone network or the mobile phone network without going through the LAN, it can be connected to the provider network without going through the LAN. The communication network includes a WAN and the Internet. The communication network may be constructed by either wired or wireless, and wired and wireless may be combined.

  The imaging device 1 may be a camera with a general angle of view (for example, a focal length of 35 mm), or a camera that captures 360 degrees around and captures an omnidirectional image. A camera that captures an angle of view between a general angle of view and 360 degrees around may also be used. However, a wider range can be covered with a smaller number of imaging devices 1 as the angle of view increases.

  The imaging device 1 may be called a digital still camera or a digital video camera. Further, when the communication terminal 3 has a camera, the communication terminal 3 can be a digital camera. In the present embodiment, in order to make the explanation easy to understand, the imaging apparatus 1 will be described as a digital camera for obtaining an omnidirectional image. The imaging device 1 periodically images around 360 degrees. The image management apparatus 5 may not necessarily be periodic, may take images irregularly, may be taken by an operation of the installer X, or may be requested from the image management apparatus 5 by the viewer Y requesting the image management apparatus 5. You may take an image with the command. Moreover, you may change the time interval imaged according to a time slot | zone. For example, the time period during which the product is easy to sell (the time period when there are many visitors) is shortened, and the time period when the product is difficult to sell (the time period when there are few visitors) is lengthened.

  Note that the imaging apparatus 1 may automatically capture several scenes with different line-of-sight directions and create a spherical image by combining a plurality of image data.

  The imaging device 1 is arranged at a place where there is a subject to be fixed point observed. Although FIG. 4 is arranged in the store, it is only an example. The use scene for moving object removal other than the EC site will be described later. The number of the imaging devices 1 varies depending on the arrangement location. In general, a small number is sufficient for an arrangement place with a good view, and a large number is provided for an arrangement place with a poor view.

  The communication terminal 3 has a communication function for connecting to the communication network 9 instead of the imaging device 1. The communication terminal 3 is a cradle for supplying power to the imaging apparatus 1 and fixing it to a store. The cradle refers to an expansion device that expands the function of the imaging device 1. The communication terminal 3 has an interface for connecting to the imaging device 1, and thus the imaging device 1 can use the function of the communication terminal 3. The communication terminal 3 performs data communication with the imaging device 1 via this interface. Then, data communication is performed with the image management apparatus 5 via the wireless router 9a and the communication network 9.

  Note that when the imaging device 1 has a function of directly performing data communication with the wireless router 9a and the communication network 9, the communication terminal 3 may be omitted. Or the imaging device 1 and the communication terminal 3 may be comprised integrally.

  The image management device 5 is an information processing device that functions as a server, for example, and can perform data communication with the communication terminal 3 and the terminal device 7 via the communication network 9. The image management device 5 performs image processing for removing moving objects from the image, and provides the image to the terminal device 7 as a Web page. Therefore, the image management apparatus 5 operates as a Web server. This web page may be used, for example, as a web page of an EC site that sells merchandise at a store. Since the live video of the store is included in the Web page, shopping with a sense of reality becomes possible. The web page may be a monitoring web page for monitoring the product shelf 81 in the store. The administrator can confirm that the products on the product shelf 81 are few and replenish them, or can confirm that the products on the product shelf 81 are disturbed and align them.

  When an omnidirectional image is captured, OpenGL ES (API: Application Interface for 3D graphics) is installed in the image management apparatus 5. By calling OpenGL ES, an equirectangular image can be created from the omnidirectional image, or a thumbnail image of a part of the omnidirectional image (predetermined area image) can be created.

  Note that cloud computing may be applied to the image management apparatus 5. Although there is no strict definition of the physical configuration of cloud computing, the configuration of the information processing device is configured by dynamically connecting and disconnecting resources such as the CPU, RAM, and storage that make up the information processing device according to the load. In addition, a configuration in which the installation location is flexibly changed is known. In cloud computing, the image management apparatus 5 is generally virtualized. One information processing apparatus can provide a function as a plurality of image management apparatuses 5 by virtualization, or a plurality of information processing apparatuses can provide a function as one image management apparatus 5 by virtualization. . Note that the image management apparatus 5 can be provided not by cloud computing but by a single information processing apparatus.

  The terminal device 7 is an information processing device that acquires and displays a Web page including an image from which a moving object has been removed from the image management device 5. For example, a PC (Personal Computer) performs data communication with the image management apparatus 5 via the communication network 9. The terminal device 7 may be a notebook PC, a tablet terminal, a PC, a PDA (Personal Digital Assistant), an electronic blackboard, a video conference terminal, a wearable PC, a game machine, a mobile phone, a car navigation system, a smartphone, or the like. Moreover, it is not restricted to these.

  The imaging device 1, the communication terminal 3, and the wireless router 9a are installed at predetermined positions by the installer X at each sales base such as a store. The terminal device 7 that communicates with the EC site may be arranged in a place where a general consumer lives or may be carried by a general consumer. The terminal device 7 that monitors a store may be arranged at a head office that supervises each store, the store manager's home, an office, a merchandise distributor, or the like, or may be carried by an administrator.

<Example of store image and imaging device arrangement>
FIG. 5 is an example of an arrangement of the image of the store 2 and the imaging device 1. In a general store 2, product shelves 81 are arranged across a passage. In many cases, drinks are displayed on a product shelf 81 having a refrigerator function near the wall. The number and arrangement of the imaging device 1 may be determined so that all the products on all the product shelves 81 are in the imaging range, or the number and arrangement are determined so that one or more predetermined products are in the imaging range. May be. When the imaging device 1 captures an omnidirectional image, a desired product can be captured with a smaller number than the imaging device 1 having a general angle of view. In FIG. 5, the imaging device 1 is arranged on the upper part of the product shelf 81 just before the wall. The imaging device 1 may be arranged on the ceiling or in the middle of the product shelf 81.

<Hardware Configuration of Embodiment>
Next, the hardware configuration of the imaging device 1, the communication terminal 3, the terminal device 7, and the image management device 5 according to the present embodiment will be described with reference to FIGS.

<< Imaging device >>
FIG. 6 is an example of a hardware configuration diagram of the imaging apparatus 1. Hereinafter, the imaging apparatus 1 is an imaging apparatus using two imaging elements, but the number of imaging elements may be three or more. Further, it is not always necessary to use an apparatus dedicated to omnidirectional imaging, and it may have substantially the same function as the imaging apparatus 1 by attaching a retrofit omnidirectional imaging unit to a normal digital camera, smartphone, or the like. .

  As shown in FIG. 6, the imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory). ) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, operation unit 115, network I / F 116, communication unit 117, and antenna 117a.

  Among these, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 103a and 103b are provided. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by a fisheye lens into image data of an electric signal and output the image data. A timing generation circuit for generating a vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103 a and 103 b of the imaging unit 101 are connected to a serial I / F bus (I2C bus or the like) separately from the imaging control unit 105. The image processing unit 104 and the imaging control unit 105 are connected to the CPU 111 via the bus 110. Further, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I / F 116, communication unit 117, and electronic compass 118 are connected to the bus 110.

  The image processing unit 104 takes in the image data output from the image sensors 103a and 103b through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. Create equirectangular data.

  In general, the imaging control unit 105 sets a command or the like in a register group of the imaging elements 103a and 103b using the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to capture status data of the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

  The imaging control unit 105 instructs the imaging elements 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some imaging devices 1 may have a preview display function using a display or a function corresponding to moving image display. In this case, output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frame / min).

  Further, as will be described later, the imaging control unit 105 also functions as a synchronization control unit that synchronizes the output timing of image data of the imaging elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the imaging device 1 is not provided with a display unit, but may be provided with a display unit.

  The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data.

  The CPU 111 controls the overall operation of the imaging apparatus 1 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data being processed, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and processed equirectangular figure data.

  The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user inputs various imaging modes, imaging conditions, and the like by operating the operation buttons.

  The network I / F 116 is a generic name for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. Further, the network I / F 116 may be a network interface regardless of wireless or wired. The data of the equirectangular figure stored in the DRAM 114 is recorded on an external medium via the network I / F 116, or the communication terminal 3 via the network I / F 116 which becomes a network I / F as necessary. Or sent to an external device.

  The communication unit 117 is connected to an external device such as the communication terminal 3 through an antenna 117a provided in the imaging device 1 by a remote radio technology such as Wi-Fi (wireless fidelity), NFC, or LTE (Long Term Evolution). Communicate. The communication unit 117 can also transmit the equirectangular cylindrical data to the external device of the communication terminal 3.

  The electronic compass 118 calculates the azimuth and tilt (Roll rotation angle) of the imaging device 1 from the earth's magnetism, and outputs azimuth / tilt information. This azimuth / tilt information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. The related information includes each data of the image capturing date and time and the data capacity of the image data.

<< Communication terminal >>
Next, the hardware configuration of the communication terminal 3 will be described with reference to FIG. FIG. 7 is a hardware configuration diagram of the communication terminal 3 in the case of a cradle having a wireless communication function.

  As shown in FIG. 7, the communication terminal 3 includes a CPU 301 that controls the operation of the entire communication terminal 3, a ROM 302 that stores basic input / output programs, and a RAM (Random Access Memory) 304 that is used as a work area for the CPU 301. , Wi-Fi (registered trademark), NFC, LTE, etc., a communication unit 305 for data communication, a USB I / F 303 for wired communication with the imaging device 1, and an RTC (Real Time Clock) 306 for holding calendar and time information have.

  In addition, a bus line 310 such as an address bus or a data bus for electrically connecting the above-described units is provided.

  The ROM 302 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

  The communication unit 305 communicates with the wireless router 9a and the like by a wireless communication signal using the antenna 305a.

  In addition to the figure, a GPS receiving unit is provided that receives GPS signals including position information (latitude, longitude, and altitude) of the communication terminal 3 by a GPS (Global Positioning Systems) satellite or IMES (Indoor Messaging System) as an indoor GPS. It may be.

<< Image management device, terminal device >>
The hardware configuration of the terminal device 7 in the case of the image management device 5 and the notebook PC will be described with reference to FIG. FIG. 8 is a hardware configuration diagram of the image management device 5 and the terminal device 7. Since both the image management device 5 and the terminal device 7 are computers, the configuration of the image management device 5 will be described below. It is assumed that the configuration of the terminal device 7 is the same as that of the image management device 5, and even if there is a difference, there is no problem with the description of the present embodiment.

  The image management apparatus 5 includes a CPU 501 that controls the operation of the entire image management apparatus 5, a ROM 502 that stores a program used to drive the CPU 501 such as an IPL, and a RAM 503 that is used as a work area for the CPU 501. Also, an HD 504 that stores various data such as a program for the image management apparatus 5 and an HDD (Hard Disk Drive) 505 that controls reading or writing of various data to the HD 504 according to the control of the CPU 501. In addition, it has a media drive 507 for controlling reading or writing (storage) of data with respect to a recording medium 506 such as a flash memory, and a display 508 for displaying various information such as a cursor, menu, window, character, or image. The display 508 is preferably equipped with a touch panel. In addition, a network I / F 509 for data communication using the communication network 9, a keyboard 511 having a plurality of keys for inputting characters, numerical values, and various instructions, selection and execution of various instructions, and a processing target A mouse 512 for selecting, moving a cursor, and the like is included. Further, it has a CD-ROM drive 514 that controls reading or writing of various data with respect to a CD-ROM (Compact Disc Read Only Memory) 513 as an example of a removable recording medium. Further, as shown in FIG. 8, a bus line 510 such as an address bus or a data bus is provided for electrically connecting the above components.

<Functions of image processing system>
FIG. 9 is a functional block diagram of the imaging device 1, the communication terminal 3, the image management device 5, and the terminal device 7 included in the image processing system 200 of the present embodiment.

<< Functional configuration of imaging device >>
The imaging apparatus 1 includes a reception unit 12, an imaging unit 13, a sound collection unit 14, a connection unit 15, and a storage / readout unit 19. Each of these units is a function realized by any of the constituent elements shown in FIG. 6 being operated by a command from the CPU 111 according to the program for the imaging device 1 developed from the SRAM 113 onto the DRAM 114, or Means.

  In addition, the imaging apparatus 1 includes a storage unit 1000 configured by one or more of the ROM 112, the SRAM 113, and the DRAM 114 illustrated in FIG. The storage unit 1000 stores a program for the imaging device 1 and a terminal ID.

  The receiving unit 12 of the imaging device 1 receives an operation input to the imaging device 1 from a user (installer X in FIG. 4). Note that the imaging device 1 automatically and periodically captures the surroundings without an operation for imaging by the installer X. The installation interval X is set in the imaging apparatus 1 by the installer X. Alternatively, the viewer Y who is an administrator of the head office may set the image via the image management apparatus 5.

  The imaging unit 13 images the surroundings and creates image data. In the present embodiment, image data of an omnidirectional image in which 360 degrees around is created is created, but image data having a general angle of view may be used.

  The sound collecting unit 14 collects sounds around the imaging device 1 and converts them into sound data. When the audio data is transmitted to the terminal device 7, the viewer Y can purchase a product from the EC site in a more realistic state. Note that since voice data may contain information with high privacy, the voice data may not be transmitted.

  The connection unit 15 receives power supply from the communication terminal 3 and performs data communication with the communication terminal 3. The power may be supplied from another power source such as an AC adapter. Further, the imaging device 1 and the communication terminal 3 may communicate wirelessly.

  The storage / reading unit 19 stores various data in the storage unit 1000 and reads various data from the storage unit 1000. In the following description, the description “via the storage / reading unit 19” may be omitted even when the imaging apparatus 1 reads / writes data from / from the storage unit 1000.

<< Functional configuration of communication terminal >>
The communication terminal 3 includes a transmission / reception unit 31, a reception unit 32, a connection unit 33, and a storage / reading unit 39. Each of these units is a function realized by any one of the constituent elements shown in FIG. 7 being operated by a command from the CPU 301 according to the program for the communication terminal 3 expanded from the ROM 302 to the RAM 304, or Means.

  Further, the communication terminal 3 includes a storage unit 3000 constructed by the ROM 302 and the RAM 304 shown in FIG. The storage unit 3000 stores a program for the communication terminal 3.

(Functional configuration of communication terminal)
The transmission / reception unit 31 of the communication terminal 3 transmits / receives various data to / from the image management apparatus 5 via the wireless router 9 a and the communication network 9. In the following description, even when the communication terminal 3 communicates with the image management apparatus 5, the description “via the transmission / reception unit 31” may be omitted.

  The connection unit 33 supplies power to the imaging device 1 and performs data communication. The imaging device 1 and the communication terminal 3 may communicate wirelessly. The communication terminal 3 is supplied with power from an AC adapter or the like.

  The storage / reading unit 39 stores various data in the storage unit 3000 and reads various data from the storage unit 3000. Hereinafter, even when the communication terminal 3 reads / writes data from / from the storage unit 3000, the description “through the storage / reading unit 39” may be omitted.

<< Functional configuration of image management device >>
The image management apparatus 5 includes a transmission / reception unit 51, an image division unit 52, a similarity calculation unit 53, a screen creation unit 54, a determination unit 55, an image composition unit 56, and a storage / readout unit 59. Each of these units is a function realized by any one of the constituent elements shown in FIG. 8 operating according to a command from the CPU 501 according to the program for the image management apparatus 5 expanded from the HD 504 onto the RAM 503. Or means.

  Further, the image management apparatus 5 includes a storage unit 5000 constructed by the RAM 503 and the HD 504 shown in FIG. In the storage unit 5000, a store management DB 5001 and a division number DB 5002 are constructed. Hereinafter, each database will be described.

表１は、店舗管理ＤＢ５００１に記憶される各情報をテーブル状に示す店舗管理テーブルを示す。店舗管理テーブルには、店舗ＩＤ、店舗名、住所、拠点レイアウトマップ、端末ＩＤと商品、及び、画角の各項目が対応付けて記憶されている。また、店舗管理テーブルの１つの行をレコードという場合がある。

Table 1 shows a store management table that shows each piece of information stored in the store management DB 5001 in the form of a table. In the store management table, store ID, store name, address, base layout map, terminal ID, product, and angle of view are stored in association with each other. One row of the store management table may be referred to as a record.

  The store ID is an example of identification information for identifying the store 2. The store ID is assigned to store 2 so as not to overlap. An example of a store ID is a combination of numbers and alphabets that do not overlap. Note that ID is an abbreviation of Identification and means an identifier or identification information. ID refers to a name, a code, a character string, a numerical value, or a combination of one or more of these used to uniquely distinguish a specific target from a plurality of targets. The same applies to the following IDs.

  The store name is the name of the store 2 and is mainly used by the viewer Y to determine the store 2. The address indicates the location of the store 2. The address is used when the image processing system 200 displays the position of the store 2 on the map. Addresses are converted to latitude and longitude as needed.

  In the store layout map, file names such as image data indicating the layout of each store are registered. The position of the imaging device 1 in the store 2 and the position of the product etc. are specified by two-dimensional coordinates by the store layout map.

  The item of terminal ID and product is an item that associates the imaging device 1 with the product. When the product is selected on the store layout map, the image management device 5 is used for specifying the imaging device 1 in the store associated with the product. The terminal ID is identification information for identifying the imaging device 1. The terminal ID is, for example, a serial number, a serial number, a numerical value that does not overlap with the model number, an IP address, a MAC address, or the like of the imaging apparatus 1, but is not limited thereto. As shown in Table 1, one store 2 is provided with one or more imaging devices 1 (terminal IDs), and the positions of products are registered in the store layout map. The product and the terminal ID do not always correspond one-to-one, and a plurality of products may be associated with one terminal ID.

  The angle of view is an angle of view in which the product that is the subject is shown. Since the 360-degree surrounding is shown in the omnidirectional image, the range in which the same type of product is shown is a part of the omnidirectional image. For this reason, the angle of view is registered in association with the product. Among the angles of view, H1 and V1 are latitude and longitude, and indicate the coordinates of the center of the product area of the same type. α1 designates an area of the same type of product with an angle of view. The angle of view is coordinate information that specifies an area of the same type of product. In addition, you may designate the area | region of the same kind of goods with a pair of diagonal vertex. Since the image processing system 200 designates the range of the angle of view of the omnidirectional image and transmits it to the terminal device 7, the terminal device 7 displays a product shelf image of a predetermined product immediately after receiving the omnidirectional image. it can.

表２は、分割数ＤＢ５００２に記憶される各情報をテーブル状に示す分割数テーブルを示す。分割数テーブルには全天球画像が分割される際、どのくらいの大きさに分割されるが登録されている。分割数テーブルは画角、分割数、及び、商品の各項目を有する。画角と商品は表１の店舗管理テーブルと同じものである。分割数は画角で指定される領域を縦と横それぞれ何個に分割するかを示す。表２では縦と横の分割数は同じであるが異ならせてもよい。分割数が多いほど分割後の画像の大きさは小さくなる。分割数で分割後の画像の大きさを指定するのでなく分割後の画像の大きさそのものを指定してもよい。

Table 2 shows a division number table that shows each piece of information stored in the division number DB 5002 in a table form. In the division number table, the size of the celestial sphere image when it is divided is registered. The division number table has items of an angle of view, the number of divisions, and a product. The angle of view and the product are the same as the store management table in Table 1. The number of divisions indicates the number of vertical and horizontal divisions of the area specified by the angle of view. In Table 2, the vertical and horizontal division numbers are the same, but may be different. The larger the number of divisions, the smaller the size of the image after division. Instead of specifying the size of the divided image by the number of divisions, the size of the divided image itself may be specified.

  By determining the size of the image after division for each product, the image management device 5 can easily distinguish changes in moving objects and product shelves, and the moving objects can be easily removed from the omnidirectional image. Details will be described with reference to FIG. Note that it is not always necessary to change the size of the divided image for each product, and the size of the divided image may be the same regardless of the type of product.

(Each functional configuration of the image management device)
The transmission / reception unit 51 of the image management device 5 transmits / receives various data to / from the communication terminal 3 or the terminal device 7 via the communication network 9. Hereinafter, even when the image management device 5 communicates with the terminal device 7, the description “through the transmission / reception unit 51” may be omitted.

  The image dividing unit 52 divides the omnidirectional image into blocks having a predetermined size. By dividing the block, it is possible to remove the moving object even if the moving object does not appear at all in one omnidirectional image. Therefore, the omnidirectional image of the store 2 with a small number of visitors may not be divided. In addition, when the image captured by the imaging apparatus 1 is not an omnidirectional image but a general angle of view, it may not be divided. The division into blocks will be described with reference to FIG.

  The similarity calculation unit 53 calculates the similarity between the output image and the input image for each image divided into blocks. Details will be described later. In order to calculate the similarity, the similarity calculation unit 53 holds the output image transmitted to the terminal device 7.

  The determination unit 55 determines whether or not to update the output image by determining whether or not the similarity is greater than or equal to a threshold value. That is, it is determined whether the input image is a subject image. When the similarity is above the threshold, it is determined to update the output image with the current input image, and when the similarity is less than the threshold, it is determined to output the output image one cycle before.

  The image composition unit 56 synthesizes the divided images determined by the determination unit 55 as the output image to generate the original omnidirectional image. When the screen creation unit 54 transmits image data to the terminal device 7, the terminal device 7 displays the spherical image using HTML data (or XHTML data), JavaScript (registered trademark), CSS (Cascade Style Sheet), or the like. Create a Web page to do this. Since the terminal device 7 analyzes the Web page and displays the screen, the Web page becomes screen information. Note that, when referred to as a web page in the present embodiment, it is assumed that a web page appropriately constructed by a web application is included. A web application refers to software used on a web browser or a mechanism thereof, which operates when a program in a script language (for example, JavaScript (registered trademark)) that operates on a web browser and a program on the web server side cooperate. .

  The storage / reading unit 59 stores various data in the storage unit 5000 and reads various data from the storage unit 5000. Hereinafter, even when the image management apparatus 5 reads and writes from the storage unit 5000, the description “through the storage / reading unit 59” may be omitted.

<Functional configuration of terminal device>
The terminal device 7 includes a transmission / reception unit 71, a reception unit 72, a display control unit 73, and a storage / readout unit 79. Each of these units is a function realized by any one of the constituent elements shown in FIG. 8 being operated by a command from the CPU 501 according to the program for the terminal device 7 developed on the RAM 503 from the HD 504 or Means.

  Further, the terminal device 7 includes a storage unit 7000 constructed by the RAM 503 and the HD 504 shown in FIG. The storage unit 7000 stores a program for the terminal device 7. The program for the terminal device 7 is, for example, browser software, but may be application software having a communication function such as browser software.

(Each functional configuration of terminal equipment)
The transmission / reception unit 71 of the terminal device 7 transmits / receives various data to / from the image management device 5 via the communication network 9. Hereinafter, even when the terminal device 7 communicates with the image management device 5, the description “through the transmission / reception unit 71” may be omitted.

  The accepting unit 72 accepts an operation input from a user (browser Y in FIG. 4). In the present embodiment, selection of the product shelf 81 (imaging device 1), selection of the product purchased by the viewer Y, reception of the number, rotation of the image, enlargement or reduction of the image in which the product is reflected, and the like are received.

  The display control unit 73 performs control for interpreting the screen information transmitted from the image management device 5 and displaying various screens on the display 508 of the terminal device 7.

  The storage / reading unit 79 stores various data in the storage unit 7000 and reads various data from the storage unit 7000. Hereinafter, even when the terminal device 7 reads / writes data from / from the storage unit 7000, the description “through the storage / reading unit 79” may be omitted.

<Image division>
Since the omnidirectional image has a wide angle of view, it may be copied if there is a moving object in the store. When the visitor frequency to the store 2 is high, the frequency of the omnidirectional image in which the moving object is reflected also increases. In the moving object removal method of outputting a past omnidirectional image in which a person 82 is not reflected in a certain image, the omnidirectional image in which the person 82 is not reflected at all is changed to a past one according to the visitor frequency. Therefore, the old product shelf 81 is displayed in time. For this reason, in this embodiment, moving object removal is performed after the omnidirectional image is divided into blocks. If divided into blocks, the frequency of moving objects appearing in one block is reduced, and an image of the product shelf 81 that is not so old can be output.

  The size of the divided image will be described with reference to FIG. FIG. 10 is an example of a diagram illustrating the size of the divided image. The size of the divided image is determined in consideration of the size of the product (drink in FIG. 10) and the size of the moving object.

  Consider a case where the size of an image after division is set to be approximately the same as that of a product as shown in FIG. The size of the divided image is indicated by a dotted line 91. In the present embodiment, it is desired to output a past output image when a moving object is shown, and to output an input image when the number of items on the item shelf 81 decreases or increases. However, if the size of the image after the division and the size of the product are approximately the same, the degree of similarity may be lowered both when a moving object is captured and when the product is reduced or increased, and thus cannot be distinguished.

  Next, consider a case where the size of the divided image is larger than that of the moving object as shown in FIG. The size of the divided image is indicated by a dotted line 91. When a moving object is shown in the divided image, the similarity is lowered. However, even when the number of products is reduced or increased, the similarity is lowered and there is a possibility that the similarity cannot be distinguished. For example, if the human body overlaps a part of the divided image, it becomes close to the degree of similarity when the product is reduced, and cannot be distinguished.

  From the above, it is preferable that the size of the image after the division is at least larger than the product and not more than the size of the moving object. Furthermore, the size of the image after division is preferably several times the size of the product and is completely covered by the moving object. By doing so, the similarity of the divided image in which the moving object is captured is sufficiently low, and the similarity when the number of products is reduced or increased is higher than that, so it is easy to distinguish both. In the division number table of Table 2, the division number is determined so that both can be easily distinguished. Although the present embodiment is merely an example, the omnidirectional image is divided into sizes into which about four products are included.

<Similarity>
The similarity is an index relating to how similar the image after the division is. It may be referred to as the degree of coincidence or the degree of correlation.

式（１）のＲは相互相関係数であり、Ｉは例えば入力画像、Ｔは出力画像であり、（ｉ、ｊ）は画素位置を示す。Ｎは横方向の画素数であり、Ｍは縦方向の画素数である。相互相関係数Ｒは１が最大で０が最小である。相互相関係数Ｒが大きいほど類似していることを意味する。

In Equation (1), R is a cross-correlation coefficient, I is, for example, an input image, T is an output image, and (i, j) indicates a pixel position. N is the number of pixels in the horizontal direction, and M is the number of pixels in the vertical direction. The cross correlation coefficient R has a maximum of 1 and a minimum of 0. It means that it is so similar that the cross correlation coefficient R is large.

  Further, the calculation similar to the equation (1) may be performed by subtracting the average pixel value of the input image from the input image and subtracting the average pixel value of the output image from the output image. As a result, the similarity can be stably calculated even if the brightness of the input image varies.

  Further, SAD (Sum of Absolute Difference) that is the sum of absolute values of pixel value differences and SSD (Sum of Squared Difference) that is the sum of squares of differences of luminance values may be used. However, although there is no change in the number of products in SAD or SSD, the value tends to increase due to changes in brightness due to illumination or sunlight. For this reason, it is preferable to obtain the similarity by a cross-correlation coefficient. However, since the lighting is always on in the store, there is little change in brightness, and SAD or SSD may be used.

  If a technique for recognizing a human face is known and the moving object is a person 82, the image management apparatus 5 may detect the moving object using this technique. However, when the imaging device 1 images the product shelf 81, a person is imaged from behind, so no face is captured. For this reason, it is difficult to detect the face of the person 82 and it is not suitable for the detection of the moving object of the present embodiment.

<Determination of output image>
A method for determining an output image based on the similarity will be described with reference to FIG. 11A shows the similarity, FIG. 11B shows the input image, and FIG. 11C shows the output image. The determination unit 55 repeats the determination of whether the similarity between the output image and the input image is greater than or equal to a threshold value.

Time t ₀ : When the imaging device 1 captures the omnidirectional image for the first time, the input image 86 — t ₀ is the output image as it is.

Time _{t 1:} Since there is no change in the product of the shelves 81, the similarity of the output image 87_T ₀ of the input image 86_T ₁ and time _{t 0} becomes higher threshold or more. Determination unit 55 determines the input image 86_T ₁ at time _{t 1} in the output image 87_t _1.

Time _{t 2:} for the person 82 is captured, the similarity of the output image 87_T ₁ of the input image 86_T ₂ and time _{t 1} is less than the threshold value. Determination unit 55 determines the output image 87_T ₁ at time _{t 1} as the output image.

Time t _{3 to} t ₅ : Since the person 82 continues to be captured, the similarity between the input images 86_t _{3 to} 86_t ₅ and the output images 87_t _{2 to} 87_t ₄ one cycle before is less than the threshold. The determination unit 55 determines the output images 87_t _{2 to} 87_t ₄ one cycle before as output images.

Time t ₆ : Since the person 82 picks up the product and leaves from the front of the product shelf 81, the product shelf 81 is shown in the input image 86 — t ₆ , but the product is reduced by one. Therefore, the similarity of the output image of the input image and the time t ₅ the time t6 is greater than if it person 82 captured smaller than circumstances without completely changes to a product. Hereinafter, this similarity is referred to as “medium similarity”. Since similarity moderate a higher threshold, determination unit 55 determines the input image 86_T ₆ of time t6 to the output image.

Time _{t n:} Then, when there is no change in the number of products, because the similarity of the input image 86_T _n of output image 87_T ₆ and time _{t n} of time _{t 6} is high (similarity approximately 1), the determination unit 55 determines an input image 86_t _n at time t _n as an output image.

As can be seen from the output images at times t _{2 to} t ₅ , when a moving object is detected from the input image, the output images 87_t _{2 to} 87_t ₅ become the output images of the past output images 87_t _{1 as} they are. Y can browse the product displayed by the terminal device 7 almost in real time (with a delay of several seconds). Further, the privacy of the person 82 in the store 2 can be protected.

  The similarity calculation unit 53 and the determination unit 55 calculate the similarity and determine the output image for each divided image. When the output image is determined from all the divided images, the image combining unit 56 combines the divided images into one omnidirectional image.

<Operation procedure>
FIG. 12 is an example of a sequence diagram illustrating an overall procedure in which the image processing system 200 provides an image. In FIG. 12, a scene is assumed in which the viewer Y uses the image management apparatus 5 as an EC site.

  S12-1: The terminal device 7 logs into the image management device 5. In a general EC site, a viewer is registered as a member, and information related to the viewer Y (address, telephone number, e-mail address, etc.) is specified by logging in. Although it is possible for the viewer Y to purchase a product without logging in, in this case, the viewer Y needs to input an address or the like.

  S12-2: For the logged-in terminal device 7, the screen creation unit 54 of the image management device 5 creates a store list screen including a list of stores 2 near the viewer Y from the address of the viewer Y. The screen creation unit 54 compares the address of the viewer Y registered in advance or the location information of the terminal device 7 transmitted from the terminal device 7 with the address of the store management table, for example, 10 stores including the nearest store 2 Specify the degree. And the store list screen which specifies the position of the store 2 on a map is created. The transmission / reception unit 51 of the image management device 5 transmits a store list screen to the terminal device 7. An example of the store list screen is shown in FIG.

  S12-3: The transmission / reception unit 71 of the terminal device 7 receives the store list screen, and the display control unit 73 displays the store list screen on the display 508. When the viewer Y selects a store 2 where he / she wants to purchase a product, the reception unit receives the selection of the product.

  S12-4: The transmission / reception unit 71 of the terminal device 7 transmits a store ID for specifying the store 2 to the image management device 5. The transmission / reception unit 51 of the image management apparatus 5 receives the store ID. Thereby, since the store 2 can be specified, the image management device 5 can transmit the product screen of the store 2 to the terminal device 7.

  S12-5: The imaging device 1 in the store 2 periodically images the surroundings as an example.

  S12-6: The imaging device 1 transmits the omnidirectional image, the store ID, and the device ID to the image management device 5 through the communication terminal 3. For convenience of explanation, the communication terminal 3 is omitted in FIG. In addition, all the imaging devices 1 in the store each transmit an omnidirectional image.

  S12-7: The transmission / reception unit 51 of the image management device 5 receives the omnidirectional image, the store ID, and the device ID, and the similarity calculation unit 53 and the determination unit 55 determine the output image. This process will be described with reference to FIG.

  S12-8: The transmission / reception unit 51 of the image management device 5 transmits the product screen including the omnidirectional image to the terminal device 7. An example of the product screen is shown in FIG.

  S12-9: The transmission / reception unit 71 of the terminal device 7 receives the product screen, and the display control unit 73 displays the product screen on the display 508. When the browsing person Y selects the product shelf 81 of the product to be purchased on the product screen, the reception unit 72 receives the selection of the product shelf 81.

  S12-10: The transmission / reception unit 71 of the terminal device 7 transmits the terminal ID associated with the product shelf 81 to the image management device 5. As a result, the image management device 5 can identify the imaging device 1 and transmit the omnidirectional image captured by the imaging device 1 to the terminal device 7.

  S12-11: The transmission / reception unit 51 of the image management device 5 transmits the omnidirectional image transmitted by the imaging device 1 specified by the terminal ID to the terminal device 7 together with the angle of view. Since 360-degree surroundings are captured in the omnidirectional image, this angle of view is used to specify the range in which the product is captured. The display control unit 73 of the terminal device 7 displays the range specified by this angle of view on the display 508.

  Note that the transmission timing of the omnidirectional image in step S12-10 is an example, and the omnidirectional image for which the determination of the output image has been completed is performed without delay regardless of whether the terminal ID is transmitted from the terminal device 7. The management device 5 may transmit to the terminal device 7. When only the omnidirectional image of the imaging device 1 designated by the terminal ID is transmitted as in step S12-10, the communication load can be reduced. When the image management device 5 transmits the omnidirectional image for which the output image has been determined to the terminal device 7 without delay, when the viewer Y selects the product shelf 81 (the omnidirectional image), Sphere image can be switched.

<Screen example>
FIG. 13 shows an example of the store list screen 601 displayed by the terminal device 7. The store list screen 601 has a map display column 602, a search column 603, a store information column 604, and a decision button 605. The map display column 602 is a column that displays a map centered on the viewer Y, and displays stores 2 around the viewer. The number of stores 2 may be determined in advance or may be set by the viewer Y. The viewer Y can enlarge or reduce the map, display more stores 2, and confirm more detailed routes to the stores 2.

  The search field 603 further has an input field 603a and a search button 603b. The input column 603a is a column for the viewer Y to input the store name and location, and the search button 603b is a button for the terminal device 7 to transmit a search request to the image management device 5. When the store 2 is searched, the map display field 602 is also updated, and a map centering on the store 2 that has been hit by the search is displayed.

  The store information column 604 is a column for displaying detailed information of the selected store 2. On the store list screen 601 immediately after transmission, detailed information of the store 2 used nearest or last is displayed. The viewer Y can select an arbitrary store 2 from the map display field 602 with a pointing device such as a mouse or a touch panel. The decision button 605 is a button for the terminal device 7 to transmit to the image management device 5 that the image of the selected store 2 is displayed. Thus, the viewer Y can select an arbitrary store 2 and select a product.

  Instead of the map display field 602 or together with the map display field 602, the list of stores 2 may be displayed as text data.

  FIG. 14 shows an example of the product screen 611 displayed by the terminal device 7. The product screen 611 includes a store layout map field 612, a product image field 613, a cart 614, and a button 615 for proceeding to a cash register. In the store layout map field 612, a store layout map is displayed. The store layout map field 612 displays a simplified map of where and what products are in the store. For example, a product category is displayed on the product shelf icon 612a, and each product shelf icon 612a also serves as a button for accepting selection by the pointing device 618. As shown in the store management table of Table 1, each merchandise shelf icon 612a is associated with the terminal ID of the imaging device 1 (linked or embedded), and when the merchandise shelf icon 612a is selected. The terminal ID of the imaging device 1 is specified. The currently selected merchandise shelf icon 612a is highlighted.

  In the product image column 613, the omnidirectional image of the imaging device 1 associated with the product shelf icon 612a selected in the store layout map is displayed. Although the angle of view immediately after the display is designated by the image management device 5, the viewer Y can rotate the omnidirectional image to display an arbitrary direction. In the omnidirectional image, the product name and the price of the product shown in the region are associated with each region occupied by one product (one individual product, not a product category). The area is specified by the angle of view. When the accepting unit 72 of the terminal device 7 accepts a click (or tap) of the pointing device 618, the display control unit 73 displays a product associated with the accepted region and its price as a pop-up image 617. The pop-up image 617 may be displayed by a simple mouse over.

  The viewer Y checks whether or not the product is a desired product, and drags and drops the pointing device 618 to the cart 614 if the product is the desired product. The receiving unit 72 detects that the product has been dragged and dropped onto the cart 614, and associates the product with the cart 614. By an operation such as clicking (or tapping) on the cart 614, the display control unit 73 displays a list of products in the cart 614, a price, a total amount, and the like.

  A button 615 for proceeding to the cash register is a button for transitioning to a screen for settlement of the cart merchandise. The viewer Y can set the delivery destination of the product or pay the total amount on this screen.

<Determination of output image>
FIG. 15 is an example of a flowchart illustrating a procedure in which the image management apparatus 5 determines an output image. The process of FIG. 15 starts every time the imaging apparatus 1 captures one omnidirectional image and transmits it to the image management apparatus 5.

  First, the image dividing unit 52 divides the omnidirectional image (step S15-1). The image dividing unit 52 refers to the division number table and divides the omnidirectional image by the division number associated with the angle of view. An angle of view that is not registered in the division number table is divided into predetermined sizes. Further, it is not necessary to divide the entire omnidirectional image. This is because it may be considered that there is no moving object near the ceiling of the store 2 or just below the imaging device 1. Therefore, the image dividing unit 52 does not have to divide the angle of view near the ceiling or the imaging device 1.

  Next, the similarity calculation unit 53 calculates the similarity between the output image one cycle before and the current input image for each divided image (step S15-2).

  Then, the determination unit 55 determines whether or not the similarity between the output image one cycle before and the current input image is equal to or greater than a threshold value (step S15-3).

  If the similarity is equal to or greater than the threshold value, the determination unit 55 updates the output image with the input image (step S15-4). If the degree of similarity is less than the threshold value, the determination unit 55 determines to output the output image one cycle before as an output image (step S15-5).

  Then, the determination unit 55 determines whether or not all the divided images have been processed (step S15-6). For the divided image corresponding to the angle of view where there is no possibility of moving objects, the output image may be updated with the input image without determining the output image. Thereby, the time required for determining the output image can be shortened.

  When all the divided images have been processed, the image composition unit 56 synthesizes the input image or the output image according to the determination result for each block by the determination unit 55 to generate one omnidirectional image (Step S1). S15-7). The screen creation unit 54 transmits a Web page including the omnidirectional image to the terminal device 7 via the transmission / reception unit 51. Alternatively, only the omnidirectional image is transmitted to the terminal device 7. The display control unit 73 of the terminal device 7 can display the omnidirectional image from which the moving object is removed on the display 508.

<Summary>
As described above, the image processing system 200 according to the present embodiment outputs a past omnidirectional image in which the person 82 is not captured if the person 82 is captured in an image at a certain time. The image of the product shelf 81 can be provided by removing. Since the output image before the moving object is detected is displayed while the moving object is captured, the viewer cannot browse the product. Even if the product on the product shelf 81 is taken out and the image on the product shelf 81 is changed, the decrease in the similarity is slight, so that the output image can be updated with the input image on the product shelf 81.

  In the present embodiment, an image processing system 200 that does not update an output image with an input image by detecting even a part of a moving object in an image after division will be described. In the present embodiment, the constituent elements denoted by the same reference numerals in the first embodiment perform similar functions, and therefore, only the main constituent elements of the present embodiment may be mainly described.

  FIG. 16 is an example of a diagram for explaining inconveniences solved by the image processing system 200 of the present embodiment. FIG. 16A shows an output image having a certain period, and FIGS. 16B and 16C show input images after time.

  In FIG. 16B, since the number of products is reduced by one, the degree of similarity becomes medium as described in the first embodiment. Since the degree of similarity is medium, the determination unit 55 determines to update the output image in FIG. 16A with the input image in FIG. On the other hand, in FIG. 16C, since a part of the person 82 (for example, a finger) is shown, the similarity is similarly medium. Since the degree of similarity is medium, the determination unit 55 determines to replace the output image in FIG. 16A with the input image in FIG. However, since a part of the person 82 is shown, the viewer feels uncomfortable.

  In order to deal with such an inconvenience, the present embodiment describes an image processing system 200 that can determine that a person 82 is reflected in an input image as shown in FIG.

<Determination of output image>
The output image determination method based on the similarity in this embodiment will be described with reference to FIG. FIG. 17A shows the similarity, FIG. 17B shows the input image, and FIG. 17C shows the output image. The determination unit 55 repeats the determination of whether the similarity between the output image and the input image is greater than or equal to a threshold value.

Time t ₀ : When the imaging device 1 captures the omnidirectional image for the first time, the input image 86 — t ₀ is the output image as it is.

Time _{t 1:} Since there is no change in the product of the shelves 81, the similarity of the output image 87_T ₀ of the input image 86_T ₁ and time _{t 0} becomes higher threshold or more. In this embodiment, when the degree of similarity is medium or higher, the determination unit 55 calculates the degree of similarity between the input image before a predetermined number of cycles and the current input image. In FIG. 17, the degrees of similarity between the input image two cycles before and four cycles before and the current input image 86_t ₁ are calculated. And if the degree of similarity is high, it is determined that the change in the shelf 81, the determination unit 55 determines the input image 86_T ₁ at time _{t 1} in the output image 87_t _1.

Time _{t 2:} for the person 82 is captured, the similarity of the output image 87_T ₁ of the input image 86_T ₂ and time _{t 1} is less than the threshold value. Determination unit 55 determines the output image 87_T ₁ at time _{t 1} as the output image.

Time t _{3 to} t ₅ : Since the person 82 continues to be captured, the similarity between the input images 86_t _{3 to} 86_t ₅ and the output images 87_t _{2 to} 87_t ₄ one cycle before is less than the threshold. The determination unit 55 determines the output images 87_t _{2 to} 87_t ₄ one cycle before as output images.

Time t ₆ : Since the person 82 picks up the product and leaves from the front of the product shelf 81, the product shelf 81 is shown in the input image, but the product is reduced by one. Therefore, the similarity of the output image 87_T ₅ of the input image 86_T ₆ and time _{t 5} the time _{t 6} becomes moderate. When the degree of similarity is medium or higher, the degree-of-similarity calculation unit 53 calculates the degree of similarity between the input images 86_t ₂ and 86_t ₄ two cycles before and four cycles before and the current input image 86_t ₆ , respectively. If the similarity between the output image and the input image is equal to or greater than the threshold value, but the similarity between the past input image and the current input image is low, it cannot be determined whether the change in the product shelf 81 or a part of the human body is reflected, determination unit 55 determines the output image 87_T ₅ time t5 of 1 cycle before it the output image.

Time tn: If there is no change in the number of products thereafter, the similarity between the output image 87_t _{6 at} time t6 and the input image 86_t _n at time tn becomes medium or higher. and the previous cycle of the input image, calculates the respective current input image 86_T _n and similarity to. When it is time after the person 82 went away, because of the high input image 86_T _n and similarity between two periods before and four cycles before the input image at time tn (because almost coincide), determination unit 55 the current input determining the image 86_T _n in the output image. Therefore, when the input image was not a while changes, the change of more than moderate output image and the input image it is determined that the change in the background, can update the output image in the input image 86_t _n.

According to the moving object removal method of the present embodiment, since the output image 87_t ₅ is not updated with the current input image 86_t ₆ at time t6, the timing to be transmitted to the terminal device 7 is delayed even if the product shelf has actually changed. End up. However, since a part of the human body is reflected even though the commodity shelf 81 has not changed, it is erroneously determined that the commodity shelf 81 has changed, and the output image is the current input image in which a part of the human body is reflected. Can be suppressed.

Supplementary description will be made with reference to FIG. Figure 18 is a view similar to FIG. 17, the input image 86_T ₆ at time _{t 6} is different from FIG. 17.

Time t ₆ : The person 82 has left the front of the merchandise shelf 81 but has not taken the merchandise. However, it is reflected part of the human body to the current input image 86_t _6. Therefore, the similarity of the output image 87_T ₅ of the input image 86_T ₆ and time _{t 5} the time _{t 6} becomes moderate. When the degree of similarity is medium or higher, the degree-of-similarity calculation unit 53 calculates the degree of similarity between the input images 86_t ₂ and 86_t ₄ two cycles before and four cycles before and the current input image 86_t ₆ , respectively.

Since the degree of similarity is low, the determination unit 55 determines the output image 87_T ₅ time t5 of 1 cycle before it the output image. Therefore, even if a part of the human body is shown, the output image is updated with the current input image showing a part of the human body even if the similarity between the input image and the output image of the previous cycle becomes medium. This can be suppressed.

  As understood from FIG. 18, the input images before 2 cycles and 4 cycles before are compared with the current input image when a part of the human body is shown in the current input image. This is because there is a high possibility that the human body is reflected in the input image several seconds before (predetermined time) in view of the moving speed. When one image is captured per second, 2 cycles before and 4 cycles before correspond to 2 seconds before and 4 seconds before. When the imaging cycle of the imaging device 1 is changed, as an example, input images of 2 seconds before and 4 seconds before may be extracted. Therefore, the timing of two cycles before and four cycles before is an example, and it is only necessary to compare the input image before a predetermined time considered from the moving speed of the human and the current input image. For example, it may be 1 cycle before, 3 cycles before, 3 seconds before, 5 seconds before, etc.

  The reason for comparing the two timings before and two cycles before is to confirm more reliably that the human body has not been captured in the past, and at either one of the timings before or after two cycles. You may compare. This is because if the similarity between one of the input images and the current input image is low, a part of the human body may be reflected in the current input image.

  The degree of similarity between one of the input images from one cycle before to five cycles before and the current input image may be calculated, or two or more of the input images from one cycle before to five cycles before (one second cycle) In this case, the similarity between a maximum of five input images and the current input image may be calculated. When the similarity between a plurality of input images and the current input image is calculated, it is determined whether all the similarities are equal to or greater than a threshold value.

<Operation procedure>
FIG. 19 is an example of a flowchart for explaining a procedure in which the image processing system 200 of the present embodiment updates the output image. In the description of FIG. 19, differences from FIG. 15 will be described. First, the processes of steps S19-1 and S19-2 may be the same as steps S15-1 and S15-2 of FIG.

  In step S19-3, the determination unit 55 determines whether the similarity is greater than or equal to a threshold value x (step S19-3). The threshold value x is a threshold value lower than the similarity when there is a change in the product shelf 81 or when a part of the human body is shown. The threshold value x is approximately the same as the threshold value in step S15-3 in FIG.

  When the determination in step S19-3 is Yes, the similarity calculation unit 53 calculates the similarity between the input image two cycles before and four cycles before and the current input image (step S19-4). That is, the degree of similarity between an input image that may include a moving object several seconds ago and the current input image is calculated.

  The determination unit 55 determines whether the similarity between the input image two cycles before and the current input image is greater than or equal to the threshold y, and whether the similarity between the input image four cycles before and the current input image is greater than or equal to the threshold y. Judgment is made (step S19-5).

  If it is determined Yes in step S19-5, since no moving object is shown in the past input image, the determination unit 55 determines to update the output image with the current input image (S19-6). Thereby, when a change is matched with the goods shelf 81, an output image can be updated with the input image after a change.

  If it is determined No in step S19-5, there is a possibility that a moving object is reflected in the past input image, and therefore it is determined whether a part of the human body is reflected in the current input image or a change in the product shelf. The determination unit 55 determines that the output image of the previous cycle is the output image as it is (S19-7). Thereby, when a part of human body is reflected in the current input image, it is possible to avoid updating the output image.

  Note that the threshold y is a value close to 1, which is the maximum similarity, because it is desired to detect that the input image matches the input image several seconds ago (the product shelf 81 is shown and has no change). Therefore, the threshold value y is larger than the threshold value x. Thereby, when the input image does not change, the output image can be updated with the input image.

  The processing in steps S19-8 and S19-9 may be the same as steps S15-6 and S15-7 in FIG.

  As described above, the image processing system 200 according to the present embodiment detects even if a part of the moving object appears in the divided image by comparing the current input image with the past input image. Updating the output image can be suppressed.

  In this embodiment, after determining whether or not a moving object is shown in the divided image, an input process in which only a part of the moving object is shown is performed by performing an extension process that extends that the output image is determined not to be updated. An image processing system 200 that more reliably suppresses updating an output image with an image will be described.

  FIG. 20 is a functional block diagram of the imaging device 1, the communication terminal 3, the image management device 5, and the terminal device 7 included in the image processing system 200 of the present embodiment. In the description of FIG. 20, the constituent elements denoted by the same reference numerals as those in FIG. 9 perform the same function, and therefore, only the main constituent elements of the present embodiment may be mainly described.

  The image management apparatus 5 according to the present exemplary embodiment includes an extension processing unit 57. The extension processing unit 57 performs processing that regards that the moving object is detected even in the divided image adjacent to the divided image in which the moving object is detected. That is, the range of the divided image in which the moving object is detected is expanded.

<Extended processing>
FIG. 21 is an example of a diagram illustrating the extension process. In FIG. 21, the square represents the image after division, the image “1” represents the image after division where the moving object is detected, and “0” represents the image after division where the moving object is not detected. First, FIG. 21A shows nine divided images that have been subjected to moving object detection processing according to the first and second embodiments. In FIG. 21A, the moving object is detected only in the center image.

  FIG. 21B is a diagram for explaining the extension that a moving object has been detected in images adjacent to the top, bottom, left, and right of the divided image from which a moving object has been detected. In FIG. 21B, “1” is set to the image adjacent to the center image in the vertical and horizontal directions. Such an extension process is called a 4-neighbor extension process.

  FIG. 21C is a diagram for explaining the extension that the moving object is detected in the vertically adjacent image and the diagonally adjacent image of the divided image where the moving object is detected. In FIG. 21C, “1” is set to an image adjacent to the center image vertically and horizontally and diagonally. Such an extension process is called an 8-neighbor extension process.

  FIG. 22 is an example of a diagram illustrating a preferred example of the 8-neighbor expansion process. As shown in FIG. 22A, for example, it is assumed that the head of a human body is reflected in the center of nine divided images. It is considered that a moving object is detected from the images at the center, the lower left, the lower center, and the lower right. As shown in FIG. 22B, in the 4-neighbor expansion process, it can be considered that the human body is shown in the upper, lower, left, and right images, but a part of the human body is also shown in the obliquely upper image. When the human body is slightly shown in the upper right and upper left images, the output image may be updated even by the determination method of the second embodiment.

  On the other hand, as shown in FIG. 22 (c), in the 8-neighbor expansion process, it can be considered that a part of the human body is shown in the upper right and upper left images, so that the output image is represented by the input image showing the moving object. Updates can be suppressed.

  Therefore, it can be seen that the 8-neighbor expansion process is effective, but the 4-neighbor expansion process may be employed depending on the shape of the moving object.

  FIG. 23 is an example of a flowchart in which the extension processing unit 57 performs the extension process on the divided image. In the process of FIG. 23, differences from FIG. 19 will be mainly described.

  In the process of FIG. 23, steps S23-1 to S23-4 are added. In step S23-1, the extension processing unit 57 determines one image after division (step S23-1). The method for determining the divided image may be arbitrary.

  Next, it is determined whether or not a moving object has been detected in the divided image of interest (step S23-2).

  When the determination in step S23-2 is Yes, the extension processing unit 57 considers that a moving object has been detected in the up / down / left / right and diagonal images (step S23-3). If the determination in step S23-2 is No, the extension processing unit 57 does not perform the extension process.

  The extension processing unit 57 determines whether or not the processing has been completed for all the divided images (step S23-4). If the processing has been completed, the expanded processing unit 57 combines the divided images (S19-9).

  It should be noted that in step S23-2, the divided image that is considered to have detected a moving object is not determined to be Yes. Thereby, it can be avoided that the divided image adjacent to the divided image where the moving object is detected and the adjacent divided image are considered to be detected in a chain.

  According to the present embodiment, by dividing the divided image that is determined not to update the output image to the vicinity, a part of the person 82 is shown, but the similarity with the past input image is equal to or greater than the threshold value y. Even in such a case, the output image is not updated, so that it is possible to prevent the output image from being updated with an input image in which only a part of the moving object is shown.

  In the present embodiment, an image processing system 200 that mechanically learns an image showing a human body, constructs a learning model 58, and determines whether or not a person is shown in an input image using the learning model 58 will be described.

  FIG. 24 is a functional block diagram of the imaging device 1, the communication terminal 3, the image management device 5, and the terminal device 7 included in the image processing system 200 of the present embodiment. In the description of FIG. 24, the components denoted by the same reference numerals as those in FIG. 9 perform the same functions, and therefore, only the main components of the present embodiment may be mainly described.

  The image management apparatus 5 of this embodiment has a learning model 58. The learning model 58 holds a learning result of an image in which a moving object is captured or an image in which a moving object is not captured, and outputs the presence / absence of a moving object with respect to the input image. The learning model 58 outputs whether or not a moving object is shown in the input image (or accuracy, probability, probability, etc.). You may use the feature-value extracted from the input image. The learning apparatus constructs a learning model 58 by learning in advance whether or not a person is shown in the input image for training using an algorithm such as deep learning, support vector machine, neural network, or random forest. This learning result is held in the learning model 58, and functions as a determination unit that determines whether or not a moving object is shown in the input image (whether it is an image of a subject).

<Judgment using learning model>
1. Discarding Input Image with Learning Model Only A method for determining an output image using the learning model 58 will be described with reference to FIG. FIG. 25A shows an input image, and FIG. 25B shows an output image. The determination unit 55 determines an output image based on whether or not a moving object is captured using the learning model 58 alone.

Time _{t 0:} When the input image 86_t ₀ is captured, it is input to the learning model 58. The learning model 58 determines that no person is captured. Since there is no output image at time t ₀ , the determination unit 55 determines to output the input image 86_t ₀ .

Time _{t 1:} the input image 86_T ₁ is imaged is inputted to the learning model 58. Since it is determined that the learning model 58 is not photographed a person, to determine the input image 86_t ₁ of time _{t 1} to the output image 87_t _1.

Time t ₂ : Since the person 82 is captured in the input image 86_t ₂ , the learning model 58 determines that the person is captured, discards the input image 86_t _2, and uses the output image 87_t ₁ at time t _{1 as} an output image as it is. decide.

Time t _{3 to} t ₅ : Since the person 82 continues to be captured, the learning model 58 determines that the person is captured. Therefore, the output images 87_t _{2 to} 87_t ₄ one cycle before are determined as output images as they are.

Time t ₆ : Since the person 82 picks up the product and leaves from the front of the product shelf 81, the product shelf 81 is shown in the input image, but the product is reduced by one. Since it is determined that the learning model 58 is not photographed a person, to determine the input image 86_t ₆ of the time _{t 6} to the output image 87_t _6.

  In this way, by using the learning model 58, it is possible to discard an input image in which a person is captured with the accuracy of the learning model 58.

  FIG. 26 is an example of a flowchart illustrating a procedure for the learning model to discard the input image. The process in FIG. 26 is repeatedly executed every time an input image is captured. First, the input image is divided into blocks (S15-1).

  The learning model determines whether or not a moving object is shown in the input image (S26-1). If no moving object is shown in the input image, the learning model updates the output image with the input image (S26-2).

  If a moving object is shown in the input image, the learning model outputs an output image one cycle before (S26-3).

  Then, the learning model 58 determines whether or not all the divided images have been processed (S15-6). If all the divided images have not been processed, the processing from step S26-1 is repeated. If all the divided images have been processed, the divided images are synthesized (S15-7).

2. The input image is discarded based on the similarity between the input image and the output image that are not discarded in the learning model. A method for determining the output image using the learning model 58 will be described with reference to FIG. FIG. 27A shows an input image, and FIG. 27B shows an output image. The determination unit 55 uses the learning model 58 and determines an output image based on whether the similarity between the output image and the input image is equal to or greater than a threshold value.

Time _{t 0:} When the input image 86_t ₀ is captured, it is input to the learning model 58. The learning model 58 determines that no person is captured. Since there is no output image at time t ₀ , the determination unit 55 determines to output the input image 86_t ₀ .

Time _{t 1:} the input image 86_T ₁ is imaged is inputted to the learning model 58. The learning model 58 determines that no person is captured. Then, the similarity calculation unit 53 calculates the degree of similarity between the input image 86_T ₁ and the output image 87_t _0. Since there is no change in the product of the shelves 81, the similarity of the input image 86_T ₁ and the output image 87_T ₀ becomes higher threshold or more. When the similarity is equal to or larger than the threshold, the determination unit 55 determines the input image 86_T ₁ at time _{t 1} in the output image 87_t _1.

Time _{t 2:} Since the input image 86_T ₂ has captured a person 82, learning model 58 determines that the person is captured. Determining unit 55 discards an input image 86_t _2, to determine the output image 87_T ₁ at time _{t 1} as the output image.

Time t _{3 to} t ₅ : Since the person 82 continues to be captured, the learning model 58 determines that the person is captured. The determination unit 55 determines the output images 87_t _{2 to} 87_t ₄ one cycle before as output images.

Time t ₆ : Since the person 82 picks up the product and leaves from the front of the product shelf 81, the product shelf 81 is shown in the input image, but the product is reduced by one. First, the learning model 58 determines that no person is captured. Similarity of the output image 87_T ₅ of the input image 86_T ₆ and time _{t 5} the time _{t 6} becomes moderate. For moderate similarity (or threshold), the determination unit 55 determines the input image 86_T ₆ at time _{t 6} to the output image 87_t _6.

  In this way, by using the learning model 58, it is possible to discard an input image in which a person is captured with the accuracy of the learning model 58. Since the similarity calculation unit 53 calculates the similarity with respect to the input image determined by the learning model 58 that the person is not captured, the input image in which the person is actually captured can be discarded.

<Operation procedure>
FIG. 28 is an example of a flowchart illustrating a procedure in which the image management apparatus 5 determines an output image. In the description of FIG. 28, differences from FIG. 15 will be mainly described.

  First, the image dividing unit 52 divides the omnidirectional image (step S15-1).

  Next, it is determined whether or not a person is shown in the image obtained by dividing the learning model 58 (S26-1). If it is determined that a person is captured, the process proceeds to step S15-5, and the determination unit 55 determines to output the output image one cycle before as an output image (step S15-5).

  If it is determined that no person is captured, the process proceeds to step S26-1 and the learning model 58 determines whether a moving object is captured for each of the divided images (step S26-1). The subsequent processing may be the same as in FIG.

<Summary>
Therefore, according to the image processing system 200 of the present embodiment, since the learning model 58 is used, an input image in which a person is captured can be detected with high accuracy if the accuracy of the learning model 58 is high. Since the image determined by the learning model 58 that the person is captured does not include the image in which the product shelf is updated, the image determined by the learning model 58 that the person is not captured is the image in which the product shelf is updated or The image is not updated. When the product shelf has been updated, if it is desired to update the output image with the input image at an early stage, the output image can be updated at an early stage in a situation where there is a low possibility that a human body is shown by reducing the threshold value.

  Further, the process using the learning model 58 can be applied in combination with any one of the second and third embodiments as well as the first embodiment. In combination with any of the embodiments, the learning model 58 may first determine whether or not a person is shown in the input image. In the third embodiment, when the learning model 58 determines that a person is captured, the extension processing unit 57 performs extension processing around the block where the person exists.

  In this embodiment, the learning model 58 determines whether or not a person is captured before the similarity between the input image and the output image is calculated. After the similarity between the input image and the output image is calculated, The learning model 58 may determine whether a person is shown. Since the similarity is higher than the threshold value, even if the determination unit 55 determines to update the output image with the input image, the input image can be discarded when a part of the human body is captured. In other words, since it is necessary to update the output image with the input image for updating the product shelf, there is a limit to increasing the threshold value (threshold value x in the second embodiment). Even if there is an input image in which the threshold value is low and the human body is captured, the learning model 58 can detect this and discard the input image in which the person is captured.

  In this embodiment, a modification of the image processing system 200 will be described.

<Device for moving object removal>
In the first to third embodiments, the image management apparatus 5 performs moving body removal. However, the moving body removal may be performed by any one of the imaging apparatus 1, the communication terminal 3, the wireless router 9a, or the terminal apparatus 7. In addition, an information processing apparatus such as a computer or a microserver present on the store side may perform the process. Further, although not shown in FIG. 4, a dedicated information processing apparatus for moving object removal connected via the communication network 9 may remove the moving object.

  In FIG. 4, the terminal device 7 acquires the omnidirectional image from which the moving object is removed from the image management device 5, but the terminal device 7 may acquire the omnidirectional image directly from the imaging device 1. In this case as well, any device of the imaging device 1, the communication terminal 3, the wireless router, or the terminal device 7 may perform moving object removal. Moreover, the imaging device 1 and the terminal device 7 may be directly connected. Further, the terminal device 7 and the imaging device 1 may be configured integrally.

  When the terminal device 7 and the imaging device 1 are integrated, the terminal device 7 transmits an omnidirectional image to the image management device 5 and acquires the omnidirectional image from which the moving object is removed from the image management device 5. Also good.

<Use scenes other than stores>
In order to share information displayed on a whiteboard or an electronic blackboard used in a meeting or a lecture with a plurality of terminal devices 7, the imaging device 1 may image the whiteboard or the electronic blackboard. In some cases, the writer may be shown at the time of imaging, but the writer can be excluded by moving object removal described in the present embodiment, and the output image can be updated by detecting a change in information displayed on the whiteboard or the electronic blackboard.

  Moreover, it can utilize suitably also about a live camera. The live camera is an image pickup apparatus 1 that is installed in a remote place to monitor a remote place image from a place where the user is present and periodically picks up the image. For example, the image management apparatus 5 can remove moving objects when installed in a weather observation spot or a sightseeing spot and an environmental customer or the like is reflected in the spherical image. The imaging device 1 may be arranged not only outdoors but indoors such as a museum.

  The moving object removal method of the present embodiment can also be applied to inventory management in which the image management device 5 replenishes merchandise by analyzing an image of the imaging device 1 disposed in a distribution warehouse or the like. In the distribution warehouse, there are workers who pick up and replenish products, so it may appear in the omnidirectional image, but this embodiment removes workers from the omnidirectional image, and further, The change of the omnidirectional image due to the decrease can be output as an output image.

  The moving object removal method of this embodiment can also be applied to abnormality monitoring in which a factory line is imaged by the imaging apparatus 1 and the image management apparatus 5 detects a line abnormality. In the factory line, workers are assembling, etc., but this embodiment removes workers from the omnidirectional image and outputs the change in the omnidirectional image due to the change in the product on the line as an output image. it can.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, in the present embodiment, the fixed point observation for imaging 360 degrees around the same place (the same effect as imaging in the same direction) has been described as an example, but the imaging device 1 may not be fixed. For example, it is possible to take an image even in a state where the user is holding. In addition, when the imaging apparatus 1 moves slowly, the similarity due to the change in the background becomes medium or higher, so that the output image can be updated with the input image.

  Further, the omnidirectional image displayed by the terminal device 7 may be stored in time series. Thereby, it is possible to reproduce the change in the product shelf as to how the product in the product shelf has changed. This is also effective when recording weather changes in addition to product shelves.

  9 and 20 shown in the above embodiments are divided according to main functions in order to facilitate understanding of the processing of the image processing system 200. However, the present invention is not limited by the way of dividing or the name of each processing unit. The image processing system 200 can also be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  Further, the functions of the image management apparatus 5 may be distributed to a plurality of server apparatuses, or the image processing system 200 may have a plurality of image management apparatuses 5.

  The transmission / reception unit 51 is an example of an image acquisition unit, the similarity calculation unit 53 is an example of a similarity calculation unit, the determination unit 55 and the learning model 58 are examples of a determination unit, and the determination unit 55 and the learning model 58 is an example of a subject image determination unit, the image division unit 52 is an example of an image division unit, the expansion processing unit 57 is an example of a processing unit, and the display control unit 73 and the transmission / reception unit 51 are examples of an output unit. . The threshold x is an example of the first threshold, the threshold y is an example of the second threshold, the similarity between the input image and the output image is an example of the first similarity, and the current input image and the past The similarity of the input image is an example of the second similarity.

DESCRIPTION OF SYMBOLS 1 Imaging device 3 Communication terminal 5 Image management device 7 Terminal device 81 Product shelf 82 Person 86 Input image 87 Output image 200 Image processing system

Japanese Patent No. 5493709

Claims (15)

Information processing device
Image acquisition means for acquiring an image captured by the imaging device as an input image;
Subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
A determination unit that determines that the input image is output when it is determined that the input image is an image of a subject, and that outputs an output image that is output to the outside when it is determined that the input image is not an image of the subject;
A program for functioning as an output unit that outputs the input image or the output image according to a determination result of the determination unit. The information processing apparatus;
Function as a similarity calculation means for calculating a first similarity between the input image acquired by the image acquisition means and the output image output to the outside,
When the first similarity is greater than or equal to a first threshold, the subject image determination means determines that the input image is an image of the subject, and the first similarity is less than the first threshold The program according to claim 1, wherein the input image is determined not to be an image of a subject. The information processing apparatus functions as a learning model that learns in advance an image in which a moving object is captured or an image in which a moving object is not captured and outputs the presence or absence of the moving object in the image,
If the learning model does not determine that a moving object is included in the input image acquired by the image acquisition unit, the subject image determination unit determines that the input image is a subject image, and the moving object appears in the input image. The program according to claim 1, wherein the subject image determination unit determines that the input image is not a subject image when the learning model determines that the input image is present. If the first similarity is greater than or equal to a first threshold,
The similarity calculation means calculates a second similarity between the current input image acquired by the image acquisition means and a past input image than the current input image,
The determination means determines that the current input image is output when the second similarity is greater than or equal to a second threshold value, and outputs to the outside when the second similarity is less than the second threshold value. The program according to claim 2, wherein it is determined that the output image is output. When the learning model does not determine that a moving object is reflected in the input image acquired by the image acquisition unit, the information processing apparatus is configured to connect the input image acquired by the image acquisition unit and the output image output to the outside. Function as a similarity calculation means for calculating the first similarity,
If the first similarity is greater than or equal to a first threshold,
The similarity calculation means calculates a second similarity between the current input image acquired by the image acquisition means and a past input image than the current input image,
The determination means determines that the current input image is output when the second similarity is greater than or equal to a second threshold value, and outputs to the outside when the second similarity is less than the second threshold value. The program according to claim 3, wherein it is determined that the output image is output. When the first similarity is less than a first threshold, or when the second similarity is less than a second threshold,
5. It is determined that a moving object that appears in front of the subject when the image capturing device exists between the subject to be imaged and the image capturing device is reflected in the current input image acquired by the image acquisition unit. Or the program of 5.   The similarity calculation unit calculates the second similarity between an input image determined a predetermined time ago based on a moving speed of a moving object and the current input image acquired by the image acquisition unit. The program according to any one of 4 to 6.   The program according to any one of claims 4 to 7, wherein the second threshold value is larger than the first threshold value. Causing the information processing apparatus to function as an image dividing unit that divides the input image acquired by the image acquiring unit into blocks;
The similarity calculation means calculates the first similarity for each block of the input image divided into blocks and the output image output,
9. The determination unit according to claim 2, wherein the determination unit determines whether to output the current input image or to output the output image output to the outside for each block. program. The image dividing means refers to size information in which information relating to the size of the block is associated with coordinate information of an image captured by the imaging device,
The program according to claim 9, wherein the input image acquired by the image acquisition unit is divided into blocks.   The size of the block is larger than a subject imaged by the imaging device, and smaller than a moving object that appears in front of the subject when the subject exists between the imaging device and the subject. The program according to 9 or 10.   The information processing apparatus is further caused to function as a processing unit that determines that the output image is output to the outside for each block adjacent to the block that is determined by the determination unit when the information processing device is output to the outside. 12. The program according to any one of 11 above.   13. The program according to claim 12, wherein the processing unit determines that the output image is output to the outside for each block adjacent to the vicinity of 8 blocks determined by the determination unit to output to the outside. Image acquisition means for acquiring an image captured by the imaging device as an input image;
Subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
A determination unit that determines that the input image is output when the input image is determined to be an image of a subject, and that determines that an output image output to the outside is output when it is determined that the input image is not an image of the subject;
An information processing apparatus comprising: an output unit configured to output the input image or the output image according to a determination result of the determination unit. An image processing system having an imaging device that periodically images the surroundings and one or more information processing devices,
Image acquisition means for acquiring an image captured by the imaging device as an input image;
Subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
A determination unit that determines that the input image is output when the input image is determined to be an image of a subject, and that determines that an output image output to the outside is output when it is determined that the input image is not an image of the subject;
An image processing system comprising: an output unit that outputs the input image or the output image according to a determination result of the determination unit.

Images