JP7124281B2 - Program, information processing device, image processing system - Google Patents

Description

The present invention relates to a program, an information processing device, and an image processing system.

There is a service in which an imaging device periodically captures images from the same location in the same direction and provides the current situation of the imaging range in real time. Surveillance cameras and consumer behavior surveys target the movement of moving bodies such as people, but there are many cases in which backgrounds other than moving bodies such as people become targets of observation.

As an example of a case in which the background is to be observed, real-time distribution of images of product shelves in physical stores for EC (Electronic Commerce) sites to terminal devices of users, such as purchasers of products, is being considered. . Since the user of the EC site can see the actual product shelf on the terminal device, he/she can enjoy shopping with a sense of realism even though it is mail-order sales. However, it is not uncommon for an image of the inside of a physical store to include a visitor in front of a product shelf. Providing an image in which a visitor is shown as it is lacks protection of privacy, and if an image in which the product is hidden by the visitor is provided to the terminal device, the user cannot view the product.

Therefore, there is known a technique for removing a moving object other than the background from a captured image (see, for example, Patent Document 1). In Japanese Patent Laid-Open No. 2004-100000, a moving object detection unit detects a frame in which a moving object is shown, and a post-detection frame in which no moving object is detected is compared with each detection frame in which a moving object is detected before the post-detection frame. a drawing frame identifying unit that identifies a detected frame in which image information is drawn from the detected frame as an update frame and identifies an area of the drawn image information; When a frame is an update frame in which image information is drawn, a synthesis processing unit that superimposes the area of the corresponding image information on the image of the corresponding area of the frame after detection is disclosed.

However, since the conventional technique is a technique for removing a moving object from time-series images for a certain period of time, there is a problem that the output image cannot be updated while the image is being captured. For example, the conventional technology is a method of deleting a moving object and synthesizing image information from the point in time when a post-detection frame in which a moving object is no longer detected is detected. The handwritten image information is not displayed.

SUMMARY OF THE INVENTION An object of the present invention is to provide a program capable of updating an output image while capturing an image.

The present invention comprises an information processing apparatus comprising: image acquiring means for acquiring an image captured by an imaging device as an input image; and subject image determining means for determining whether or not the input image acquired by the image acquiring means is an image of a subject. and determining means for determining to output the input image when determining that the input image is the image of the subject, and determining to output the externally output image when determining that the input image is not the image of the subject. , output means for outputting the input image or the output image according to the determination result of the determination means; and a first degree of similarity between the input image acquired by the image acquisition means and the output image outputted to the outside When the first similarity is equal to or greater than a first threshold, the subject image determination means determines that the input image is the image of the subject, and determines that the input image is the image of the subject. If one degree of similarity is less than the first threshold, the input image is determined not to be an image of a subject; calculating a second degree of similarity between the current input image acquired by the image acquisition means and an input image past the current input image; In the above case, there is provided a program for determining to output the current input image, and determining to output the output image that was output to the outside when the second similarity is less than the second threshold .

It is possible to provide a program capable of updating an image to be output while imaging.

FIG. 10 is an example of a diagram for explaining an image of a product shelf imaged in time series and a process of removing a moving object; It is an example of the figure explaining how the image imaged periodically changes. FIG. 10 is an example of a diagram for explaining time-series input images and output images output therefrom; 1 is an example of a schematic configuration diagram of an image processing system; FIG. It is a figure explaining an example of an image of a store, and an example of arrangement of an imaging device. It is an example of the hardware block diagram of an imaging device. 1 is an example of a hardware configuration diagram of a communication terminal in the case of a cradle having a wireless communication function; FIG. 1 is an example of a hardware configuration diagram of an image management device and a terminal device; FIG. 1 is an example of functional block diagrams of an imaging device, a communication terminal, an image management device, and a terminal device included in an image processing system; FIG. It is an example of the figure explaining the size of the image after division. FIG. 10 is an example of a diagram illustrating a method of determining an output image based on similarity; FIG. 10 is an example of a sequence diagram showing an overall procedure for providing an image by the image processing system; It is a figure which shows an example of the shop list screen which the terminal device displayed. It is a figure which shows an example of the goods screen which the terminal device displayed. FIG. 10 is an example of a flowchart showing a procedure for an image management device to determine an output image; FIG. 10 is an example of a diagram for explaining a problem solved by the image processing system; FIG. 10 is an example of a diagram illustrating a method of determining an output image based on similarity (Example 2); FIG. 10 is an example of a diagram illustrating a method of determining an output image based on similarity (Example 2); FIG. 10 is an example of a flowchart illustrating a procedure for updating an output image by an image processing system (Embodiment 2); FIG. 11 is an example of functional block diagrams of an imaging device, a communication terminal, an image management device, and a terminal device, which are included in an image processing system (Embodiment 3). It is an example of the figure explaining an expansion process. It is an example of a diagram for explaining a preferred example of 8-neighbor expansion processing. FIG. 10 is an example of a flow chart in which an extension processing unit performs extension processing on a divided image; FIG. 11 is an example of functional block diagrams of an imaging device, a communication terminal, an image management device, and a terminal device, which an image processing system has (Embodiment 4). FIG. 10 is an example of a diagram illustrating a method of determining an output image using a learning model; FIG. 10 is an example of a flowchart diagram showing a procedure for a learning model to discard an input image; FIG. 10 is an example of a diagram illustrating a method of determining an output image using a learning model; FIG. 11 is an example of a flowchart showing a procedure for an image management apparatus to determine an output image (Embodiment 4).

EMBODIMENT OF THE INVENTION Hereafter, an Example is given and demonstrated, referring drawings for the form for implementing this invention.

<Comparison example of image processing for removing a moving object from an image>
Before describing the present embodiment, a comparative example of removing a moving object from an image will be described.

FIG. 1 is an example of a diagram illustrating an image of a product shelf 81 captured in time series and a moving object removal process. The imaging device is arranged at a position for imaging at least part of the product shelf, and periodically images the product shelf. Such imaging by the imaging device 1 at the same place is sometimes called fixed-point observation. The imaging time interval is a time interval at which an image can be provided early without providing an old image indefinitely when the product shelf changes, and is determined in consideration of the speed at which the product sells. Such video distribution is called real-time distribution or live distribution. In determining the time interval, the bandwidth of the network and the processing load such as image processing are taken into consideration. The shorter the time interval, the higher the real-time performance, but if the product shelf does not change in such a short time, the processing load will also increase. From the above, although the time interval is set to 1 second in this embodiment as an example only, imaging may be performed at a short time interval such as 30 frames/sec, or may be performed at 2 to 10 frames/sec. , 1 minute or more may be set as the time interval.

FIG. 1(a) shows images of a product shelf 81 captured in time series. The images 83_1, 83_2, 83_4, and 83_5 are unchanged, but the person 82 is shown in the image 83_3. When the image processing system provides the image 83_3, the person 82 is removed. As a method of removing the person 82, a method of averaging pixels of images for a predetermined time in the past to create an average image is conceivable. . If the residence time of the person 82 is 1 second and the predetermined time is 30 seconds, the effect of the image 83_3 on the average image is 1/30, so an average image without the person 82 is obtained.

However, a person 82 in the store stays in front of the same product shelf 81 for at least several seconds to look for products rather than simply passing by. As a result, the semi-transparent person 82 appears in the average image. FIG. 1(b) is a diagram schematically showing an average image 84 in which the translucent person 82 is shown. More specifically, the semi-transparent person 82 gradually becomes darker over time as the ratio of the images showing the person 82 increases, and the semi-transparent person 82 becomes lighter over time as the ratio decreases. becoming. If the number of images used for averaging is increased, the person 82 can be removed, but it also delays the reflection of changes in products in the images.

In this way, the removal of a moving object using the average image may be effective if the number of images containing the person 82 is smaller than the number of images serving as the parameter of the average image. In-store may not be suitable.

<Example of change in image focused on by the present embodiment>
Therefore, in the present embodiment, a moving object such as the person 82 is detected by comparing an output image that is actually output and a captured input image, and if a moving object is detected, a past output image in which the moving object has not been detected is used. A moving object is removed by the output moving object removal method. An output image is an image provided or displayed to a user, and an input image is a captured image.

FIG. 2 is an example of a diagram for explaining how images captured periodically change, and FIG. 3 is a diagram for explaining time-series input images and output images output from them. An example. FIG. 2 shows a change in an image in which a product shelf 81 on which drinks are placed is periodically captured. Note that the image is processed while being divided into blocks, and the division into blocks will be described later.

FIG. 2 shows the degree of similarity 85 between an output image and an input image over time (a value between 0 and 1 indicates a higher degree of similarity as the value increases) and an example of images at that time.

Time t ₀ : An image of the product shelf 81 is captured, and an input image 86 showing products (hereinafter referred to as time t ₀ to t _n for distinction) is obtained. The output image is the input image _{86_t0} .

Time t ₁ : An input image 86 — t ₁ showing the product is obtained, but there is no change from the previous input image 86 — t ₀ and the similarity is approximately 1. The output image is the input image _{86_t1} .

Time t2: Since the person 82 enters the imaging range, an input image _{86_t2} in which the person 82 is _captured is obtained. In addition, the output image at time t1 is not updated because the similarity has greatly decreased. This processing will be described with reference to FIG.

Time t ₃ : An input image 86 — t ₃ in which the person 82 is captured is obtained since the person 82 is continuously captured. In addition, the output image at time t1 is not updated because the similarity has greatly decreased.

Time t ₄ : An input image 86 — t ₄ in which the person 82 is captured is obtained because the person 82 is continuously captured. _In addition, the output image at time t1 is not updated because the similarity has greatly decreased.

Time t ₅ : Even if the part of the person 82 to be imaged changes, the person 82 is still being imaged, so an input image 86 — t ₅ in which the person 82 is captured is obtained. _The similarity fluctuates but remains low, and the output image at time t1 is not updated.

Time t ₆ : When the person 82 leaves the imaging range, an input image 86 — t ₆ showing the product shelf 81 is captured. Since the degree of similarity increases, the output image is updated to the input image _{86_t6} . However, the similarity is slightly less than ₁ because there is one item missing at time t6.

FIG. 3(a) shows the degree of similarity 85 with respect to time, FIG. 3(b) shows input images _{86_t 0} to _{86_t 6} , and FIG. to distinguish). Since the image to be provided by the image processing system of this embodiment is the product shelf 81, the input images 86_t ₂ to 86_t ₄ in which the person 82 is shown should be removed. The image to be removed is determined from the similarity 85 being less than the threshold.

The image processing system provides past output images in place of the removed input images 86_t ₂ -86_t ₄ . The past output image is, for example, the newest output image (the newest output image _{87_t1} whose similarity is equal to or greater than the threshold).

As described above, the image processing system of this embodiment outputs a past output image in which the person 82 is not shown when the person 82 is shown in an input image. Therefore, it is possible to provide an image of the product shelf 81 as the background by removing the input image including a moving object such as the person 82 . Since the output image before the detection of the moving object is displayed while the moving object is captured, the viewer will not be unable to browse the product. Even if the product on the product shelf 81 is taken out and the image of the product shelf 81 changes, the similarity is only slightly reduced, so the output image can be updated with the input image of the product shelf with fewer products.

<Terms>
An input image is an image input to an information processing apparatus regardless of whether it is output or not. It may be an image stored in a storage medium as well as an image received from an imaging device.

An output image is an image that is output to the outside, and all or part of the input image is the output image. The output destination may be a display device such as a display, an information processing device connected via a network, or a storage medium. Output by the output means includes output to a display device, transmission to an external device, and storage in a storage medium.

A moving object is an object that has the ability to move. For example, there are humans, animals, cars/bicycles, carts (wheelbarrows), and the like. Objects that move along with human movements are also moving objects. For example, a handbag is a moving object, and a drink held by a person is also a moving object. On the other hand, a drink as a product is not a moving object because it does not have a function to move and is not held by a person.

A subject is an object to be photographed. What is the subject depends on the purpose of the imaging and can be changed. An object between the object and the imaging device may be called a moving object. A subject is also called an object to be photographed.

A background image is an image of a product shelf or an image that does not include a moving object. Also, the subject may be included in the background image such as a product shelf, or the background image may be an image of the subject.

<System configuration example>
FIG. 4 is an example of a schematic configuration diagram of the image processing system 200. As shown in FIG. The image processing system 200 has an image management device 5 , an imaging device 1 , a communication terminal 3 and a terminal device 7 which are connected via a communication network 9 . The imaging device 1 is installed by the installer X in the store. The terminal device 7 is operated by the viewer Y. FIG.

The communication network 9 is constructed to include at least one of a LAN of a store or company to which the viewer Y belongs, a provider network of a provider that connects the LAN to the Internet, and a line provided by a line operator. there is When the communication terminal 3 or the terminal device 7 is directly connected to the line telephone network or mobile phone network without going through the LAN, it can be connected to the provider network without going through the LAN. Also, communication networks include WANs and the Internet. A communication network may be constructed as either wired or wireless, or may be a combination of wired and wireless.

The imaging device 1 may be a camera with a general angle of view (for example, a focal length of 35 mm) or a camera that captures 360-degree surroundings in one shot to create a omnidirectional image. A camera that captures an angle of view between a general angle of view and 360 degrees around the circumference may also be used. However, the larger the angle of view, the smaller the number of imaging devices 1 that can cover a wider range.

The imaging device 1 is sometimes called a digital still camera or a digital video camera. Also, when the communication terminal 3 is equipped with a camera, the communication terminal 3 can be a digital camera. In the present embodiment, the imaging apparatus 1 is described as a digital camera for obtaining an omnidirectional image for the sake of easy understanding. The imaging device 1 periodically captures 360-degree images of the surroundings. It does not necessarily have to be periodic, and the image may be captured irregularly. You may image with the command of Also, the time interval for imaging may be changed depending on the time zone. For example, the time period during which the product is easy to sell (the time period when there are many visitors) is shortened, and the time period when the product is difficult to sell (the time period when there are few visitors) is lengthened.

Note that the imaging device 1 may automatically capture a number of landscapes in different line-of-sight directions and synthesize a plurality of image data to create an omnidirectional image.

The imaging device 1 is arranged at a place where there is a subject to be fixed-point observation. In FIG. 4, it is arranged in the store, but it is only an example. Usage scenes of moving object removal other than EC sites will be described later. The number of imaging devices 1 varies depending on the location. In general, a small number is sufficient for locations with good visibility, and a large number for locations with poor visibility.

The communication terminal 3 has a communication function to connect to the communication network 9 instead of the imaging device 1 . The communication terminal 3 is a cradle for supplying power to the imaging device 1 and fixing it to a store. A cradle is an expansion device that expands the functions of the imaging device 1 . The communication terminal 3 has an interface for connecting with the imaging device 1 , so that the imaging device 1 can use the functions of the communication terminal 3 . The communication terminal 3 performs data communication with the imaging device 1 via this interface. Data communication is performed with the image manager 5 via the wireless router 9 a and the communication network 9 .

If the imaging device 1 has a function of performing data communication directly with the wireless router 9a or the communication network 9, the communication terminal 3 may be omitted. Alternatively, the imaging device 1 and the communication terminal 3 may be configured integrally.

The image management device 5 is, for example, an information processing device that functions as a server, and can perform data communication with the communication terminal 3 and the terminal device 7 via the communication network 9 . The image management device 5 performs image processing for removing a moving object from the image, and provides this image to the terminal device 7 as a web page. Therefore, the image management device 5 operates as a web server. This web page may be used, for example, as a web page of an EC site that sells products in a store. Since the web page includes a live image of the store, shopping with a sense of presence becomes possible. Also, the Web page may be a monitoring Web page for monitoring the product shelf 81 in the store. The manager can confirm that the merchandise on the merchandise shelf 81 is low and replenish the merchandise, or confirm that the merchandise on the merchandise shelf 81 is disordered and arrange the merchandise.

When an omnidirectional image is captured, OpenGL ES (API for 3D graphics: Application Interface) is installed in the image management device 5 . By calling OpenGL ES, it is possible to create an equirectangular image from an omnidirectional image, or create a thumbnail image of a part of the omnidirectional image (predetermined area image).

Note that cloud computing may be applied to the image management device 5 . Although there is no strict definition for the physical configuration of cloud computing, the configuration of an information processing device by dynamically connecting and disconnecting resources such as the CPU, RAM, and storage that make up the information processing device according to the load. and a configuration in which the installation location is flexibly changed is known. In cloud computing, the image manager 5 is generally virtualized. One information processing apparatus can provide functions as a plurality of image management apparatuses 5 by virtualization, and a plurality of information processing apparatuses can provide functions as one image management apparatus 5 by virtualization. . Note that the image management device 5 may be provided by a single information processing device instead of cloud computing.

The terminal device 7 is an information processing device that acquires from the image management device 5 a web page including an image from which the moving object has been removed and displays the web page. For example, it is a PC (Personal Computer), and performs data communication with the image manager 5 via the communication network 9 . The terminal device 7 may be a notebook PC, a tablet terminal, a PC, a PDA (Personal Digital Assistant), an electronic blackboard, a video conference terminal, a wearable PC, a game machine, a mobile phone, a car navigation system, a smart phone, or the like. Moreover, it is not restricted to these.

The imaging device 1, the communication terminal 3, and the wireless router 9a are installed at predetermined positions by an installer X at each sales base such as a store. The terminal device 7 that communicates with the EC site may be placed at a place where general consumers live, or may be carried by general consumers. The terminal device 7 that monitors the store may be placed at the head office that supervises each store, the store manager's home, office, product delivery person, or the like, or may be carried by the manager.

<Image of store and example of arrangement of imaging device>
FIG. 5 is an example of an image of the store 2 and an example of an arrangement of the imaging devices 1 . In a general store 2, product shelves 81 are arranged across an aisle. In many cases, drinks are displayed on a product shelf 81 that functions as a refrigerator along the wall. The number and arrangement of the imaging devices 1 may be determined so that all products on all product shelves 81 are within the imaging range, or the number and arrangement are determined so that one or more predetermined products are within the imaging range. may be When the imaging device 1 captures an omnidirectional image, a desired product can be captured with a smaller number than the imaging device 1 with a general angle of view. In FIG. 5, the imaging device 1 is arranged on the upper part of the product shelf 81 one before the wall. The imaging device 1 may be placed on the ceiling, or may be placed in the middle of the product shelf 81 .

<Hardware Configuration of Embodiment>
Next, hardware configurations of the imaging device 1, the communication terminal 3, the terminal device 7 and the image management device 5 of this embodiment will be described with reference to FIGS. 6 to 8. FIG.

<<Imaging device>>
FIG. 6 is an example of a hardware configuration diagram of the imaging device 1. As shown in FIG. In the following description, the imaging device 1 is assumed to be an imaging device using two imaging elements, but the number of imaging elements may be three or more. Further, the device does not necessarily need to be dedicated to omnidirectional imaging, and may have substantially the same functions as the imaging device 1 by attaching an omnidirectional imaging unit to an ordinary digital camera, smartphone, or the like. .

As shown in FIG. 6, the imaging apparatus 1 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory). ) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, a network I/F 116, a communication unit 117, and an antenna 117a.

Among them, the imaging unit 101 includes wide-angle lenses (so-called fish-eye lenses) 102a and 102b each having an angle of view of 180° or more for forming a hemispherical image, and two imaging units provided corresponding to each wide-angle lens. It has elements 103a and 103b. The imaging elements 103a and 103b are image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors and CCD (Charge Coupled Device) sensors that convert an optical image by a fisheye lens into image data of an electrical signal and output the image data. It has a timing generation circuit that generates vertical synchronization signals, pixel clocks, and the like, and a group of registers in which various commands and parameters required for the operation of this imaging device are set.

The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I/F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected separately from the imaging control unit 105 by a serial I/F bus (I2C bus or the like). The image processing unit 104 and imaging control unit 105 are connected to the CPU 111 via the bus 110 . Furthermore, ROM 112, SRAM 113, DRAM 114, operation unit 115, network I/F 116, communication unit 117, electronic compass 118, and the like are also connected to bus 110. FIG.

The image processing unit 104 fetches the image data output from the imaging elements 103a and 103b through the parallel I/F bus, performs predetermined processing on each image data, and synthesizes these image data. , to create equirectangular plot data.

The imaging control unit 105 generally uses the I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices to set commands and the like in registers of the imaging elements 103a and 103b. Necessary commands and the like are received from the CPU 111 . The imaging control unit 105 also uses the I2C bus to take in status data and the like of the registers of the imaging elements 103 a and 103 b and send them to the CPU 111 .

Further, the imaging control unit 105 instructs the imaging devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. Some imaging devices 1 have a preview display function on a display or a function corresponding to moving image display. In this case, the image data is output continuously from the imaging devices 103a and 103b at a predetermined frame rate (frames/minute).

In addition, as will be described later, the imaging control unit 105 also functions as synchronization control means for synchronizing the output timing of the image data of the imaging devices 103a and 103b in cooperation with the CPU 111. FIG. Note that in the present embodiment, the imaging apparatus 1 is not provided with a display section, but may be provided with a display section.

The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in sound data output from the microphone 108 through the I/F bus and performs predetermined processing on the sound data.

The CPU 111 controls the overall operation of the imaging device 1 and executes necessary processing. ROM 112 stores various programs for CPU 111 . The SRAM 113 and DRAM 114 are work memories, and store programs to be executed by the CPU 111, data during processing, and the like. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and data of the processed equirectangular view.

The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. A user inputs various imaging modes, imaging conditions, and the like by operating operation buttons.

The network I/F 116 is a general term for interface circuits (such as USB I/F) with external media such as SD cards and personal computers. Also, the network I/F 116 may be a network interface regardless of whether it is wireless or wired. The data of the equirectangular diagram stored in the DRAM 114 is recorded on an external medium via this network I/F 116, or is transmitted to the communication terminal 3 via the network I/F 116 which becomes a network I/F as necessary. or sent to an external device such as

The communication unit 117 communicates with an external device such as the communication terminal 3 via an antenna 117a provided in the imaging device 1 by a remote wireless technology such as Wi-Fi (wireless fidelity), NFC, or LTE (Long Term Evolution). communicate. This communication unit 117 can also transmit the equirectangular diagram data to the external device of the communication terminal 3 .

The electronic compass 118 calculates the azimuth and tilt (Roll rotation angle) of the imaging device 1 from the magnetism of the earth, and outputs the azimuth and tilt information. This azimuth/tilt information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of captured images. The related information also includes data such as the date and time when the image was captured and the data volume of the image data.

<<communication terminal>>
Next, the hardware configuration of the communication terminal 3 will be explained using FIG. Note that FIG. 7 is a hardware configuration diagram of the communication terminal 3 in the case of a cradle having a wireless communication function.

As shown in FIG. 7, the communication terminal 3 includes a CPU 301 that controls the overall operation of the communication terminal 3, a ROM 302 that stores basic input/output programs, and a RAM (Random Access Memory) 304 that is used as a work area for the CPU 301. , Wi-Fi (registered trademark), NFC, LTE, etc., a communication unit 305 for data communication, a USB I/F 303 for wired communication with the imaging apparatus 1, and an RTC (Real Time Clock) 306 for holding calendar and time information. have.

It also has a bus line 310 such as an address bus and a data bus for electrically connecting the above units.

The ROM 302 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

The communication unit 305 uses an antenna 305a to communicate with the wireless router 9a or the like using a wireless communication signal.

In addition to the illustration, a GPS receiver for receiving a GPS signal containing position information (latitude, longitude, and altitude) of the communication terminal 3 from a GPS (Global Positioning Systems) satellite or IMES (Indoor Messaging System) as an indoor GPS is provided. may be

<<Image management device, terminal device>>
The hardware configuration of the image management device 5 and the terminal device 7 in the case of a notebook PC will be described with reference to FIG. 8 is a hardware configuration diagram of the image management device 5 and the terminal device 7. As shown in FIG. Since both the image management device 5 and the terminal device 7 are computers, the configuration of the image management device 5 will be described below. It is assumed that the configuration of the terminal device 7 is the same as that of the image management device 5, and even if there is a difference, it does not hinder the explanation of the present embodiment.

The image management apparatus 5 has a CPU 501 that controls the overall operation of the image management apparatus 5 , a ROM 502 that stores programs used to drive the CPU 501 such as IPL, and a RAM 503 that is used as a work area for the CPU 501 . It also has an HD 504 that stores various data such as programs for the image management apparatus 5 and an HDD (Hard Disk Drive) 505 that controls reading and writing of various data to and from the HD 504 under the control of the CPU 501 . It also has a media drive 507 that controls reading or writing (storage) of data to a recording medium 506 such as a flash memory, and a display 508 that displays various information such as cursors, menus, windows, characters, and images. A touch panel is preferably attached to the display 508 . In addition, a network I/F 509 for data communication using the communication network 9, a keyboard 511 having a plurality of keys for inputting characters, numerical values, various instructions, etc., selection and execution of various instructions, processing target It has a mouse 512 for selection, cursor movement, and the like. It also has a CD-ROM drive 514 that controls the reading and writing of various data to and from a CD-ROM (Compact Disc Read Only Memory) 513 as an example of a removable recording medium. In addition, as shown in FIG. 8, bus lines 510 such as an address bus and a data bus are provided for electrically connecting the components described above.

<Functions of the image processing system>
FIG. 9 is a functional block diagram of the imaging device 1, communication terminal 3, image management device 5, and terminal device 7 included in the image processing system 200 of this embodiment.

<<Function configuration of imaging device>>
The imaging device 1 has a reception unit 12 , an imaging unit 13 , a sound collection unit 14 , a connection unit 15 , and a storage/readout unit 19 . Each of these units is a function or a function realized by any of the components shown in FIG. It is a means.

The imaging apparatus 1 also has a storage unit 1000 configured by one or more of the ROM 112, SRAM 113, and DRAM 114 shown in FIG. A program for the imaging apparatus 1 and a terminal ID are stored in the storage unit 1000 .

The reception unit 12 of the imaging device 1 receives an operation input to the imaging device 1 from a user (installer X in FIG. 4). Note that the imaging device 1 automatically and periodically captures images of the surroundings even if the installation person X does not perform an operation for imaging. The period interval is set in the imaging device 1 by the installer X. FIG. Alternatively, viewer Y, who is an administrator at the head office, may set via the image management device 5 .

The image capturing unit 13 captures an image of the surroundings and creates image data. In this embodiment, the image data of the omnidirectional image showing the surrounding 360 degrees is created, but the image data of a general angle of view may be used.

The sound collector 14 collects sounds around the imaging device 1 and converts them into audio data. When the audio data is transmitted to the terminal device 7, the viewer Y can purchase the product from the EC site in a more realistic state. In addition, since there is a possibility that information with high privacy is included in the voice data, the voice data does not have to be transmitted.

The connection unit 15 receives power supply from the communication terminal 3 and performs data communication with the communication terminal 3 . Power may be supplied from another power source such as an AC adapter. Also, the imaging device 1 and the communication terminal 3 may communicate wirelessly.

The storage/readout unit 19 stores various data in the storage unit 1000 and reads out various data from the storage unit 1000 . Note that, hereinafter, even when the imaging apparatus 1 reads and writes from the storage unit 1000, the description “via the storage/readout unit 19” may be omitted.

<<Function configuration of communication terminal>>
The communication terminal 3 has a transmission/reception section 31 , a reception section 32 , a connection section 33 , and a storage/readout section 39 . Each of these units is a function realized by any of the components shown in FIG. It is a means.

The communication terminal 3 also has a storage unit 3000 constructed by the ROM 302 and RAM 304 shown in FIG. A program for the communication terminal 3 is stored in the storage unit 3000 .

(Each functional configuration of the communication terminal)
The transmission/reception unit 31 of the communication terminal 3 transmits/receives various data to/from the image manager 5 via the wireless router 9 a and the communication network 9 . In the following description, even when the communication terminal 3 communicates with the image manager 5, the description "via the transmission/reception unit 31" may be omitted.

The connection unit 33 supplies power to the imaging device 1 and performs data communication. The imaging device 1 and the communication terminal 3 may communicate wirelessly. Power is supplied to the communication terminal 3 from an AC adapter or the like.

The storage/readout unit 39 stores various data in the storage unit 3000 and reads out various data from the storage unit 3000 . In addition, hereinafter, even when the communication terminal 3 reads and writes from the storage unit 3000, the description “via the storage/readout unit 39” may be omitted.

<<Function configuration of the image management device>>
The image management device 5 has a transmission/reception unit 51 , an image division unit 52 , a similarity calculation unit 53 , a screen generation unit 54 , a judgment unit 55 , an image synthesis unit 56 and a storage/readout unit 59 . Each of these units is a function realized by any one of the components shown in FIG. or means.

The image management apparatus 5 also has a storage unit 5000 constructed by the RAM 503 and the HD 504 shown in FIG. Store management DB 5001 and division number DB 5002 are constructed in this storage unit 5000 . Each database will be described below.

表１は、店舗管理ＤＢ５００１に記憶される各情報をテーブル状に示す店舗管理テーブルを示す。店舗管理テーブルには、店舗ＩＤ、店舗名、住所、拠点レイアウトマップ、端末ＩＤと商品、及び、画角の各項目が対応付けて記憶されている。また、店舗管理テーブルの１つの行をレコードという場合がある。

Table 1 shows a store management table showing each piece of information stored in the store management DB 5001 in tabular form. In the store management table, items such as store ID, store name, address, site layout map, terminal ID and product, and angle of view are stored in association with each other. Also, one row of the store management table may be called a record.

A store ID is an example of identification information for identifying the store 2 . Store IDs are assigned to stores 2 so as not to overlap. An example of the shop ID is a unique combination of numbers and letters. Note that ID is an abbreviation for Identification and means an identifier or identification information. An ID is a name, a code, a character string, a numerical value, or a combination of one or more of these used to uniquely distinguish a specific object from a plurality of objects. The same applies to the following IDs.

The store name is the name of the store 2 and is mainly used by the viewer Y to identify the store 2 . The address indicates the location of the store 2 . The address is used when the image processing system 200 displays the location of the store 2 on the map. Addresses are converted to latitude and longitude if necessary.

File names of image data indicating the layout of each store are registered in the store layout map. The position of the imaging device 1 and the positions of the products in the store 2 are specified by two-dimensional coordinates based on the store layout map.

The item of terminal ID and product is an item that associates the imaging device 1 with a product. This is used by the image management device 5 to identify the imaging device 1 in the store associated with the product when the product is selected on the store layout map. The terminal ID is identification information for identifying the imaging device 1 . The terminal ID is, for example, a serial number, manufacturing number, numerical value that does not overlap with the model number, IP address, or MAC address of the imaging device 1, but is not limited to these. As shown in Table 1, one or more imaging devices 1 (terminal IDs) are installed in one store 2, and product positions are registered in the store layout map. Products and terminal IDs do not always correspond one-to-one, and a plurality of products may be associated with one terminal ID.

The angle of view is the angle of view in which the product, which is the subject, is shown. Since the omnidirectional image captures the surrounding 360 degrees, the range in which the same type of product is captured is part of the omnidirectional image. Therefore, the angle of view is registered in association with the product. Of the angles of view, H1 and V1 are latitude and longitude, and indicate the coordinates of the center of the area of the same type of product. α1 designates the area of the same type of product by the angle of view. The angle of view is coordinate information specifying the area of products of the same type. It should be noted that a pair of diagonal vertices may be used to specify areas of the same type of product. Since the image processing system 200 designates the range of this field angle of the omnidirectional image and transmits it to the terminal device 7, the terminal device 7 displays the image of the product shelf of the predetermined product immediately after receiving the omnidirectional image. can.

表２は、分割数ＤＢ５００２に記憶される各情報をテーブル状に示す分割数テーブルを示す。分割数テーブルには全天球画像が分割される際、どのくらいの大きさに分割されるが登録されている。分割数テーブルは画角、分割数、及び、商品の各項目を有する。画角と商品は表１の店舗管理テーブルと同じものである。分割数は画角で指定される領域を縦と横それぞれ何個に分割するかを示す。表２では縦と横の分割数は同じであるが異ならせてもよい。分割数が多いほど分割後の画像の大きさは小さくなる。分割数で分割後の画像の大きさを指定するのでなく分割後の画像の大きさそのものを指定してもよい。

Table 2 shows a division number table showing each piece of information stored in the division number DB 5002 in tabular form. The number of divisions table registers how large the omnidirectional image is to be divided into divisions. The division number table has items of angle of view, division number, and product. The angle of view and products are the same as those in the store management table of Table 1. The number of divisions indicates how many vertical and horizontal divisions the area specified by the angle of view is to be divided. In Table 2, the numbers of vertical and horizontal divisions are the same, but they may be different. The larger the number of divisions, the smaller the size of the image after division. Instead of designating the size of the image after division by the number of divisions, the size of the image after division itself may be designated.

Determining the size of the image after division for each product makes it easier for the image management device 5 to distinguish between a moving object and changes in product shelves, and to remove the moving object from the omnidirectional image. Details will be described with reference to FIG. Note that it is not necessary to change the size of the image after division for each product, and the size of the image after division may be the same even if the types of products are different.

(Each functional configuration of the image management device)
The transmission/reception unit 51 of the image manager 5 transmits/receives various data to/from the communication terminal 3 or the terminal device 7 via the communication network 9 . In the following, even when the image management device 5 communicates with the terminal device 7, the description “via the transmission/reception unit 51” may be omitted.

The image dividing unit 52 divides the omnidirectional image into blocks of a predetermined size. By dividing the image into blocks, it is possible to remove the moving object even if the moving object does not appear at all in one omnidirectional image. Therefore, the omnidirectional image of the store 2 with a small number of visitors may not need to be divided. Also, if the image captured by the imaging device 1 is not an omnidirectional image but has a general angle of view, it may not be divided. The division into blocks will be explained in FIG.

The similarity calculation unit 53 calculates the similarity between the output image and the input image described above for each image divided into blocks. Details will be described later. In order to calculate the similarity, the similarity calculator 53 holds the output image transmitted to the terminal device 7 .

The determination unit 55 determines whether or not to update the output image by determining whether the degree of similarity is greater than or equal to the threshold. That is, it is determined whether or not the input image is the image of the subject. If the similarity is above the threshold, it is determined to update the output image with the current input image.

The image synthesizing unit 56 synthesizes the divided images determined by the determining unit 55 as the output image to generate the original omnidirectional image. When the image data is transmitted to the terminal device 7, the screen creation unit 54 causes the terminal device 7 to display the omnidirectional image using HTML data (or XHTML data), JavaScript (registered trademark), CSS (Cascade Style Sheet), or the like. Create a web page for Since the terminal device 7 analyzes the web page and displays the screen, the web page becomes screen information. Note that the term "web page" in the present embodiment includes a web page appropriately constructed by a web application. A web application is software or its mechanism that operates through cooperation between a script language (such as JavaScript (registered trademark)) program that runs on a web browser and a program on the web server side, and that is used on the web browser. .

The storage/readout unit 59 stores various data in the storage unit 5000 and reads out various data from the storage unit 5000 . In the following, even when the image management device 5 reads and writes from the storage unit 5000, the description “via the storage/readout unit 59” may be omitted.

<Functional configuration of terminal device>
The terminal device 7 has a transmission/reception section 71 , a reception section 72 , a display control section 73 , and a storage/readout section 79 . Each of these units is a function realized by any of the components shown in FIG. It is a means.

The terminal device 7 also has a storage unit 7000 constructed by the RAM 503 and the HD 504 shown in FIG. A program for the terminal device 7 is stored in the storage unit 7000 . The program for the terminal device 7 is, for example, browser software, but may be application software with a communication function such as browser software.

(Each functional configuration of the terminal device)
A transmission/reception unit 71 of the terminal device 7 transmits and receives various data to and from the image management device 5 via the communication network 9 . In the following, even when the terminal device 7 communicates with the image management device 5, the description “via the transmission/reception unit 71” may be omitted.

The receiving unit 72 receives an operation input from a user (viewer Y in FIG. 4). In this embodiment, selection of the product shelf 81 (imaging device 1), selection of products to be purchased by the viewer Y, acceptance of the number of products, rotation of the image, enlargement or reduction of the image showing the product, and the like are accepted.

The display control unit 73 interprets the screen information transmitted from the image management device 5 and performs control for displaying various screens on the display 508 of the terminal device 7 .

The storage/readout unit 79 stores various data in the storage unit 7000 and reads out various data from the storage unit 7000 . In the following description, even when the terminal device 7 reads and writes from the storage unit 7000, the description “via the storage/readout unit 79” may be omitted.

<Split image>
Since the spherical image has a wide angle of view, there is a possibility that if there is a moving object in the store, it will be captured. When the frequency of visits to the store 2 is high, the frequency of omnidirectional images showing a moving object also increases. In the moving object removal method of outputting a past omnidirectional image in which the person 82 is not captured when the person 82 is captured in a certain image, the omnidirectional image in which the person 82 is not captured at all is removed from the past according to the visitor frequency. Therefore, the product shelf 81 that is older in terms of time is displayed. For this reason, in the present embodiment, moving object removal is performed after the omnidirectional image is divided into blocks. If the image is divided into blocks, the frequency with which a moving object appears in one block is reduced, and an image of the product shelf 81 that is not so old can be output.

The size of the image after division will be described with reference to FIG. FIG. 10 is an example of a diagram explaining the size of an image after division. The size of the divided image is determined in consideration of the size of the product (drink in FIG. 10) and the size of the moving object.

Consider a case where the size of the image after division is set to the same size as the product, as shown in FIG. 10(a). A dotted line 91 indicates the size of the image after division. In this embodiment, it is desired to output the past output image when a moving object is captured, and to output the input image when the number of products on the product shelf 81 has decreased or increased. However, if the size of the divided image and the size of the product are about the same, there is a possibility that the degree of similarity will be low when a moving object is shown and when the number of products is reduced or increased, making it impossible to distinguish between them.

Next, as shown in FIG. 10B, consider the case where the size of the divided image is larger than that of the moving object. A dotted line 91 indicates the size of the image after division. If the divided image contains a moving object, the degree of similarity decreases, but if the number of products decreases or increases, the degree of similarity also decreases, and there is a possibility that the products cannot be distinguished from each other. For example, if a human body overlaps a part of the image after division, the similarity is close to that when the product is reduced, and it cannot be distinguished.

From the above, it is preferable that the size of the image after division is at least larger than the product and equal to or smaller than the size of the moving object. Furthermore, preferably, the size of the image after division is several times or more the size of the product, and is completely covered by the moving object. By doing this, the similarity of the divided images showing the moving object is sufficiently low, and the similarity is higher when the number of products is decreased or increased, so that both can be easily distinguished. In the division number table of Table 2, division numbers are determined so that both can be easily distinguished. Although this embodiment is merely an example, the omnidirectional image is divided into a size that can contain about four products.

<Similarity>
The degree of similarity is an index of how similar an image after division is to another image. It may also be called the degree of matching or the degree of correlation.

式（１）のＲは相互相関係数であり、Ｉは例えば入力画像、Ｔは出力画像であり、（ｉ、ｊ）は画素位置を示す。Ｎは横方向の画素数であり、Ｍは縦方向の画素数である。相互相関係数Ｒは１が最大で０が最小である。相互相関係数Ｒが大きいほど類似していることを意味する。

R in equation (1) is a cross-correlation coefficient, I is, for example, an input image, T is an output image, and (i, j) indicates a pixel position. N is the number of pixels in the horizontal direction and M is the number of pixels in the vertical direction. The cross-correlation coefficient R has a maximum value of 1 and a minimum value of 0. A larger cross-correlation coefficient R means more similarity.

Furthermore, the average of the pixel values of the input image may be subtracted from the input image, and the average of the pixel values of the output image may be subtracted from the output image to perform calculations similar to Equation (1). As a result, the similarity can be stably calculated even if the brightness of the input image fluctuates.

Alternatively, SAD (Sum of Absolute Difference), which is the sum of absolute values of differences in pixel values, and SSD (Sum of Squared Difference), which is the sum of squares of differences in luminance values, may be used. However, with SAD or SSD, even though the number of products does not change, the value tends to increase due to changes in brightness due to lighting or sunlight. Therefore, it is preferable to obtain the degree of similarity from the cross-correlation coefficient. However, since the lights are always on in the store, there is little change in brightness, so SAD or SSD may be used.

A technique for recognizing a person's face is known, and if the moving object is the person 82, it is conceivable that the image management device 5 uses this technique to detect the moving object. However, when the imaging device 1 captures an image of the product shelf 81, the face of the person is not captured because the image is captured from behind. For this reason, it is difficult to detect the face of the person 82, and is not suitable for detecting a moving object in this embodiment.

<Determination of output image>
A method of determining an output image based on the degree of similarity will be described with reference to FIG. FIG. 11(a) shows the degree of similarity, FIG. 11(b) shows the input image, and FIG. 11(c) shows the output image. The determination unit 55 repeatedly determines whether or not the degree of similarity between the output image and the input image is equal to or greater than the threshold.

Time t ₀ : When the imaging device 1 captures an omnidirectional image for the first time, the input image 86 — t ₀ is the output image as it is.

Time t ₁ : Since there is no change in the products on the product shelf 81, the similarity between the input image 86_t ₁ and the output image 87_t ₀ at time t ₀ is high and equal to or greater than the threshold. _The determination unit 55 determines the input image _{86_t1} at time t1 as the output image _{87_t1} .

Time t2: Since the person 82 is captured, the similarity between the input image _{86_t2} and the output image _{87_t1} at time t1 is _{less than} the threshold. _The determination unit 55 determines the output image _{87_t1} at time t1 as the output image as it is.

Time t ₃ to t ₅ : Since the person 82 continues to appear, the degree of similarity between the input images 86_t ₃ to 86_t ₅ and the output images 87_t ₂ to 87_t ₄ one cycle earlier is less than the threshold. The determination unit 55 determines the output images 87 — t ₂ to 87 — t ₄ of one period before as the output images as they are.

Time t ₆ : Since the person 82 picked up the product and left in front of the product shelf 81 , the product shelf 81 is shown in the input image 86_t ₆ but there is one less product. For this reason, the degree of similarity between the input image at time _t6 and the output image at time t5 is lower than when there is no change in the product and higher than when the person 82 is shown. Hereinafter, this degree of similarity will be referred to as “moderate degree of similarity”. Since the intermediate degree of similarity is equal to or higher than the threshold, the determination unit 55 determines the input image _{86_t6} at time t6 as the output image.

Time _tn : After that, _if there is no change in the number of products, the similarity between the output image _{87_t6} at time t6 and the input image _{86_tn} at time tn increases ₍ similarity is approximately 1). 55 determines the input image _{86_tn} at time _tn as the output image.

As can be seen from the output images at times t ₂ to t ₅ , when a moving object is detected from the input image, the output images 87_t ₂ to 87_t ₅ are the past output images 87_t ₁ as they are. Y can browse the products displayed by the terminal device 7 almost in real time (with a delay of about several seconds). Also, the privacy of the person 82 in the store 2 can be protected.

The similarity calculation unit 53 and the determination unit 55 calculate the similarity and determine the output image for each divided image. Then, when the output image is determined for all divided images, the image synthesizing unit 56 synthesizes the divided images into one omnidirectional image.

<Operation procedure>
FIG. 12 is an example of a sequence diagram showing an overall procedure for image processing system 200 to provide an image. In FIG. 12, a scene is assumed in which the viewer Y uses the image management device 5 as an EC site.

S12-1: The terminal device 7 logs into the image management device 5. FIG. In a typical EC site, a viewer is registered as a member, and information (address, telephone number, e-mail address, etc.) about the viewer Y is specified by logging in. It is also possible for the viewer Y to purchase the product without logging in, but in this case the viewer Y needs to enter his/her address and the like.

S12-2: The screen creation unit 54 of the image management device 5 creates a store list screen including a list of stores 2 near the viewer Y from the viewer Y's address for the logged-in terminal device 7 . The screen creation unit 54 compares the pre-registered address of the viewer Y or the position information of the terminal device 7 transmitted from the terminal device 7 with the address of the store management table, and finds, for example, 10 stores including the nearest store 2. Specify the extent. Then, a store list screen that clearly indicates the location of the store 2 on the map is created. The transmission/reception unit 51 of the image management device 5 transmits the shop list screen to the terminal device 7 . An example of the store list screen is shown in FIG.

S12-3: The transmitting/receiving unit 71 of the terminal device 7 receives the store list screen, and the display control unit 73 displays the store list screen on the display 508. FIG. When the viewer Y selects the store 2 where he/she wants to purchase the product, the reception section receives the selection of the product.

S12-4: The transmitting/receiving section 71 of the terminal device 7 transmits the shop ID for specifying the shop 2 to the image management device 5. FIG. The transmitter/receiver 51 of the image manager 5 receives the store ID. As a result, the store 2 has been specified, so the image management device 5 can transmit the product screen of this store 2 to the terminal device 7 .

S12-5: As an example, the imaging device 1 of the store 2 periodically images the surroundings.

S<b>12 - 6 : The imaging device 1 transmits the omnidirectional image, the store ID and the device ID to the image management device 5 via the communication terminal 3 . For convenience of explanation, the communication terminal 3 is omitted in FIG. Also, all the imaging devices 1 in the store transmit omnidirectional images.

S12-7: The transmission/reception unit 51 of the image management device 5 receives the omnidirectional image, the store ID, and the device ID, and the similarity calculation unit 53 and determination unit 55 determine the output image. This processing will be described with reference to FIG.

S12-8: The transmission/reception unit 51 of the image management device 5 transmits the product screen including the omnidirectional image to the terminal device 7 . An example of the product screen is shown in FIG.

S12-9: The transmitting/receiving unit 71 of the terminal device 7 receives the product screen, and the display control unit 73 displays the product screen on the display 508. FIG. When the viewer Y selects the product shelf 81 of the product to be purchased on the product screen, the reception unit 72 receives the selection of the product shelf 81 .

S12-10: The transmitting/receiving section 71 of the terminal device 7 transmits the terminal ID associated with the product shelf 81 to the image management device 5 . As a result, the image management device 5 can specify the imaging device 1 and transmit the omnidirectional image captured by the imaging device 1 to the terminal device 7 .

S12-11: The transmission/reception unit 51 of the image management device 5 transmits the omnidirectional image transmitted by the imaging device 1 identified by the terminal ID to the terminal device 7 together with the angle of view. Since the omnidirectional image captures the surrounding 360 degrees, this angle of view is used to specify the range in which the product is captured. The display control unit 73 of the terminal device 7 displays the range designated by this angle of view on the display 508 .

Note that the transmission timing of the omnidirectional image in step S12-10 is an example, and regardless of whether the terminal ID has been transmitted from the terminal device 7, the omnidirectional image for which the output image has been determined can be displayed without delay. The management device 5 may transmit to the terminal device 7 . When only the omnidirectional image of the imaging device 1 specified by the terminal ID is transmitted as in step S12-10, the communication load can be reduced. When the image management device 5 transmits the omnidirectional image for which the output image has been determined to the terminal device 7 without delay, when the viewer Y selects the product shelf 81 (omnidirectional image), the omnidirectional image is displayed in a short time. You can switch between sphere images.

<Screen example>
FIG. 13 shows an example of a store list screen 601 displayed by the terminal device 7. As shown in FIG. The store list screen 601 has a map display column 602 , a search column 603 , a store information column 604 and an enter button 605 . A map display column 602 is a column for displaying a map centered on the viewer Y, and displays stores 2 around the viewer. The number of stores 2 may be determined in advance, or may be set by the viewer Y. The viewer Y can enlarge or reduce the map, display more stores 2, and confirm more detailed routes to the stores 2. FIG.

The search field 603 further has an input field 603a and a search button 603b. An input field 603a is a field for the viewer Y to input a store name and location, and a search button 603b is a button for the terminal device 7 to transmit a search request to the image management device 5. FIG. When the store 2 is searched, the map display field 602 is also updated, and a map centering on the store 2 hit by the search is displayed.

A store information column 604 is a column for displaying detailed information on the selected store 2 . The store list screen 601 immediately after the transmission displays detailed information on the nearest or last used store 2 . The viewer Y can select an arbitrary store 2 from the map display field 602 with a pointing device such as a mouse or a touch panel. The decision button 605 is a button for the terminal device 7 to transmit to the image management device 5 that the image of the shop 2 being selected is to be displayed. In this way, the viewer Y can select an arbitrary store 2 and select a product.

Instead of the map display column 602 or together with the map display column 602, the list of stores 2 may be displayed as text data.

FIG. 14 shows an example of a product screen 611 displayed by the terminal device 7. As shown in FIG. The product screen 611 has a store layout map column 612 , a product image column 613 , a cart 614 , and a checkout button 615 . A store layout map is displayed in the store layout map column 612 . A store layout map column 612 displays a simplified map showing which products are located where in the store. For example, product shelf icons 612 a display categories of products, and each product shelf icon 612 a also serves as a button for accepting selection by the pointing device 618 . As shown in the store management table of Table 1, each product shelf icon 612a is associated (linked or embedded) with the terminal ID of the imaging device 1, and when the product shelf icon 612a is selected, A terminal ID of the imaging device 1 is specified. The product shelf icon 612a being selected is highlighted.

The product image column 613 displays the omnidirectional image of the imaging device 1 associated with the product shelf icon 612a selected on the store layout map. The angle of view immediately after being displayed is designated by the image management device 5, but the viewer Y can rotate the omnidirectional image to display any direction. In the omnidirectional image, for each area occupied by one product (one individual product, not a product category), the product name and the price of the product appearing in the area are associated with each other. A region is specified by an angle of view. When the accepting unit 72 of the terminal device 7 accepts the click (or tap) of the pointing device 618 , the display control unit 73 displays the product and its price associated with the accepted area in a pop-up image 617 . A pop-up image 617 may be displayed by a simple mouseover.

Viewer Y confirms whether or not the product is the desired product, and drags and drops the pointing device 618 onto the cart 614 if the product is the desired product. The reception unit 72 detects that the product has been dragged and dropped onto the cart 614 and associates the product with the cart 614 . When the cart 614 is clicked (or tapped) or otherwise operated, the display control unit 73 displays the list of products in the cart 614, the price, the total amount, and the like.

A checkout button 615 is a button for transitioning to a screen for settling the items in the cart. The viewer Y can set the delivery destination of the product and pay the total amount on this screen.

<Determination of output image>
FIG. 15 is an example of a flowchart showing a procedure for the image management device 5 to determine an output image. The processing in FIG. 15 starts each time the imaging device 1 captures one omnidirectional image and transmits it to the image management device 5 .

First, the image dividing unit 52 divides the omnidirectional image (step S15-1). The image dividing unit 52 refers to the division number table and divides the omnidirectional image by the division number associated with the angle of view. Angles of view not registered in the division number table are divided into predetermined sizes. Also, it is not necessary to divide the entire omnidirectional image. This is because it can be assumed that there is no moving object on the ceiling of the store 2 or in the vicinity directly below the imaging device 1 . Therefore, the image dividing unit 52 does not need to divide the angle of view near the ceiling or directly below the imaging device 1 .

Next, the similarity calculator 53 calculates the similarity between the output image of one period before and the current input image for each divided image (step S15-2).

Then, the determination unit 55 determines whether or not the degree of similarity between the output image one period before and the current input image is equal to or greater than a threshold value (step S15-3).

If the degree of similarity is greater than or equal to the threshold, the determination unit 55 updates the output image with the input image (step S15-4). If the degree of similarity is less than the threshold, the judgment unit 55 determines to output the output image of one period before as it is (step S15-5).

Then, the determination unit 55 determines whether or not the processing of all divided images has been completed (step S15-6). The output image may be updated with the input image without determining the output image for the divided image corresponding to the angle of view in which there is no possibility of capturing a moving object. As a result, the time required for determining the output image can be shortened.

When all divided images have been processed, the image composition unit 56 synthesizes the input image or the output image according to the determination result of each block by the determination unit 55 to generate one omnidirectional image (step S15-7). The screen creation unit 54 transmits the web page including the omnidirectional image to the terminal device 7 via the transmission/reception unit 51 . Alternatively, only the omnidirectional image is transmitted to the terminal device 7 . The display control unit 73 of the terminal device 7 can display the omnidirectional image from which the moving object is removed on the display 508 .

<Summary>
As described above, the image processing system 200 of the present embodiment outputs a past omnidirectional image in which the person 82 is not shown if the person 82 is shown in the image at a certain time. It can be removed to provide an image of shelf 81 . Since the output image before the detection of the moving object is displayed while the moving object is captured, the viewer will not be unable to browse the product. Even if the product on the product shelf 81 is taken out and the image on the product shelf 81 changes, the degree of similarity decreases only slightly, so the output image can be updated with the input image on the product shelf 81 .

In the present embodiment, an image processing system 200 that does not update the output image with the input image by detecting a part of the moving object even if it appears in the divided image will be described. In this embodiment, since the constituent elements denoted by the same reference numerals as in Embodiment 1 perform the same functions, only the main constituent elements of this embodiment may be mainly described.

FIG. 16 is an example of a diagram for explaining the problem solved by the image processing system 200 of this embodiment. FIG. 16(a) is an output image in a certain cycle, and FIGS. 16(b) and 16(c) are input images later in time.

In FIG. 16B, the number of products is reduced by one, so the degree of similarity is medium as described in the first embodiment. Since the degree of similarity is medium, the determination unit 55 determines to update the output image of FIG. 16(a) with the input image of FIG. 16(b). On the other hand, in FIG. 16C, part of the person 82 (for example, a finger) is shown, so the degree of similarity is also medium. Since the degree of similarity is medium, the determination unit 55 determines to replace the output image of FIG. 16(a) with the input image of FIG. 16(c). However, since a part of the person 82 is shown, the viewer feels uncomfortable.

In order to deal with such an inconvenience, this embodiment will explain an image processing system 200 that can determine that a person 82 is shown in an input image as shown in FIG. 16(c).

<Determination of output image>
A method of determining an output image based on the degree of similarity in this embodiment will be described with reference to FIG. FIG. 17(a) shows the degree of similarity, FIG. 17(b) shows the input image, and FIG. 17(c) shows the output image. The determination unit 55 repeatedly determines whether or not the degree of similarity between the output image and the input image is equal to or greater than the threshold.

Time t ₀ : When the imaging device 1 captures an omnidirectional image for the first time, the input image 86 — t ₀ is the output image as it is.

Time t ₁ : Since there is no change in the products on the product shelf 81, the similarity between the input image 86_t ₁ and the output image 87_t ₀ at time t ₀ is high and equal to or greater than the threshold. In this embodiment, when the degree of similarity is medium or higher, the determination unit 55 calculates the degree of similarity between the input image a predetermined number of cycles ago and the current input image. In FIG. 17, the degrees of similarity between the input images two cycles ago and four cycles _ago and the current input image 86_t1 are calculated. When the degree of similarity is high, it is determined that there is _a change in the product shelf 81, and the determination unit 55 determines the input image _{86_t1} at time t1 as the output image _{87_t1} .

Time t2: Since the person 82 is captured, the similarity between the input image _{86_t2} and the output image _{87_t1} at time t1 is _{less than} the threshold. _The determination unit 55 determines the output image _{87_t1} at time t1 as the output image as it is.

Time t ₃ to t ₅ : Since the person 82 continues to appear, the degree of similarity between the input images 86_t ₃ to 86_t ₅ and the output images 87_t ₂ to 87_t ₄ one cycle earlier is less than the threshold. The determination unit 55 determines the output images 87 — t ₂ to 87 — t ₄ of one period before as the output images as they are.

Time t ₆ : Since the person 82 picked up the product and left the front of the product shelf 81, the product shelf 81 is shown in the input image, but the number of products is one less. Therefore, the degree of similarity between the input image _{86_t6} at time _t6 and the output image _{87_t5} at time _t5 is moderate. When the similarity is medium or higher, the similarity calculation unit 53 calculates similarities between the input images 86_t ₂ and 86_t ₄ two cycles ago and four cycles ago and the current input image 86_t ₆ . If the degree of similarity between the past input image and the current input image is low even though the degree of similarity between the output image and the input image is greater than or equal to the threshold, it is assumed that it is impossible to determine whether the change in the product shelf 81 or a part of the human body is captured. The determination unit 55 determines the output image _{87_t5} at time t5 one cycle before as the output image as it is.

Time tn: After that, if there is no change in the number of products, the degree of similarity between the output image _{87_t6} at time t6 and the input image _{86_tn} at time tn is medium or higher. The degree of similarity between the input image before the cycle and the current input image _{86_tn} is calculated. At the timing after the person 82 has left, the similarity between the input image _{86_tn} at time tn and the input images 2 cycles and 4 cycles before is high (because they almost match). The image _{86_tn} is determined as the output image. Therefore, when the input image does not change for a while, a medium or more change between the output image and the input image can be determined as a change in the background, and the output image can be updated with the input image _{86_tn} .

According to the moving object removal method of this embodiment, the output image _{87_t5} is not updated with the current input image _{86_t6} at time t6, so the timing of transmission to the terminal device 7 is delayed even if the product shelf has actually changed. end up However, since a part of the human body appears in the product shelf 81 even though the product shelf 81 has not changed, it is erroneously determined that there is a change in the product shelf 81. can be suppressed.

A supplementary explanation is given using FIG. FIG. ₁₈ is similar to FIG. 17, but the input image _{86_t6} at time t6 is different from FIG.

Time t ₆ : The person 82 left the product shelf 81 but did not pick up the product. However, the current input image _{86_t6} shows a part of the human body. Therefore, the degree of similarity between the input image _{86_t6} at time _t6 and the output image _{87_t5} at time _t5 is moderate. When the similarity is medium or higher, the similarity calculation unit 53 calculates similarities between the input images 86_t ₂ and 86_t ₄ two cycles ago and four cycles ago and the current input image 86_t ₆ .

Since the degree of similarity is low, the determination unit 55 determines the output image _{87_t5} at time t5 one cycle earlier as the output image as it is. Therefore, even if a part of the human body is shown and the degree of similarity between the input image and the output image of one period before is intermediate, the output image is updated with the current input image that shows a part of the human body. can be suppressed.

As can be understood from FIG. 18, the reason why the current input image is compared with the input image two cycles ago and four cycles ago is that if the current input image contains a part of the human body, the This is because there is a high possibility that the human body is also captured in the input image several seconds (predetermined time) before, considering the moving speed. When one image is captured every second, two cycles before and four cycles before correspond to two seconds before and four seconds before. When the imaging cycle of the imaging device 1 is changed, as an example, it is sufficient to extract the input images two seconds and four seconds before. Therefore, the timing of two cycles before and four cycles before is just an example, and the current input image may be compared with the input image a predetermined time ago, which can be considered from the human movement speed. For example, it may be 1 cycle before and 3 cycles before, 3 seconds before and 5 seconds before, and the like.

The reason why the two timings before 2 cycles and 4 cycles before are compared is to more reliably confirm that the human body was not captured in the past. You can compare. This is because if the degree of similarity between one of the input images and the current input image is low, the current input image may include a part of the human body.

The degree of similarity between one of the input images from 1 cycle to 5 cycles ago and the current input image may be calculated, or two or more of the input images from 1 cycle to 5 cycles before The degree of similarity between the input image and the current input image may be calculated for up to five images in this case. When similarities between a plurality of input images and the current input image are calculated, it is determined whether all similarities are equal to or greater than a threshold.

<Operation procedure>
FIG. 19 is an example of a flow chart illustrating a procedure for updating an output image by the image processing system 200 of this embodiment. In the description of FIG. 19, differences from FIG. 15 will be described. First, the processing of steps S19-1 and S19-2 may be the same as steps S15-1 and S15-2 of FIG.

At step S19-3, the determination unit 55 determines whether or not the degree of similarity is equal to or greater than the threshold value x (step S19-3). The threshold x is a threshold lower than the degree of similarity when there is a change in the product shelf 81 or when a part of the human body is shown. The threshold x becomes approximately the same as the threshold in step S15-3 of FIG.

If the determination in step S19-3 is Yes, the similarity calculator 53 calculates the similarity between the input images two cycles ago and four cycles ago and the current input image (step S19-4). That is, the degree of similarity between the current input image and the input image that may include a moving object several seconds ago is calculated.

The determination unit 55 determines whether the degree of similarity between the input image two cycles ago and the current input image is equal to or greater than a threshold y, and whether the degree of similarity between the input image four cycles ago and the current input image is equal to or greater than the threshold y. decision (step S19-5).

If it is determined Yes in step S19-5, the past input image does not include a moving object, so the determining unit 55 determines to update the output image with the current input image (S19-6). As a result, when the product shelf 81 is changed, the output image can be updated with the changed input image.

If it is determined No in step S19-5, there is a possibility that the past input image contains a moving object, so it is determined whether the current input image contains a part of the human body or a change in the product shelf. Assuming that it is not possible, the determination unit 55 determines the output image of one period before as the output image as it is (S19-7). This makes it possible to avoid updating the output image when a part of the human body is shown in the current input image.

Note that the threshold value y is a value close to 1, which is the maximum degree of similarity, because it is desired to detect a match with the input image several seconds before (the product shelf 81 is shown and there is no change). Therefore, threshold y is greater than threshold x. This allows the output image to be updated with the input image when the input image does not change.

The processing of steps S19-8 and S19-9 may be the same as steps S15-6 and S15-7 of FIG.

As described above, the image processing system 200 according to the present embodiment compares the current input image and the past input image, thereby detecting a part of the moving object even if it appears in the divided image. It is possible to suppress updating the output image.

In this embodiment, after determining whether or not a moving object appears in the image after division, expansion processing is performed to expand the fact that it is determined that the output image is not to be updated. An image processing system 200 that more reliably suppresses updating an output image with an image will be described.

FIG. 20 is a functional block diagram of the imaging device 1, communication terminal 3, image management device 5, and terminal device 7 included in the image processing system 200 of this embodiment. In the description of FIG. 20, since the components denoted by the same reference numerals as in FIG. 9 perform the same functions, only the main components of this embodiment may be mainly described.

The image management device 5 of this embodiment has an extended processing section 57 . The extension processing unit 57 performs processing for determining that a moving object is also detected in a divided image adjacent to the divided image in which the moving object is detected. That is, the range of the divided image in which the moving object is detected is expanded.

<Extended processing>
FIG. 21 is an example of a diagram for explaining expansion processing. In FIG. 21, each square represents an image after division, an image of "1" represents an image after division in which a moving object is detected, and an image of "0" represents an image after division in which no moving object is detected. First, FIG. 21(a) shows nine divided images after the moving object detection processing has been performed in the first and second embodiments. In FIG. 21(a), a moving object is detected only in the central image.

FIG. 21(b) is a diagram for explaining extension of detection of a moving object to images adjacent to each other vertically and horizontally after the divided image in which the moving object is detected. In FIG. 21B, "1" is set for the images adjacent to the upper, lower, left, and right sides of the central image. Such expansion processing is called 4-neighbor expansion processing.

FIG. 21(c) is a diagram for explaining extension of detection of a moving object to an image adjacent to each other vertically, horizontally, and obliquely after the divided image in which the moving object is detected. In FIG. 21(c), "1" is set for the images adjacent to the center image in the upper, lower, left, right, and oblique directions. Such expansion processing is called 8-neighbor expansion processing.

FIG. 22 is an example of a diagram for explaining a preferred example of the 8-neighbor expansion process. As shown in FIG. 22A, for example, it is assumed that the head of a human body appears in the center of nine divided images. A moving object is considered to be detected from the center, lower left, lower center, and lower right images. As shown in FIG. 22(b), in the 4-neighbor expansion process, it can be considered that the human body is shown in the upper, lower, left, and right images, but a part of the human body is also shown in the diagonally upper image. If the upper right and upper left images only slightly show a human body, there is a possibility that the output image will be updated even with the determination method of the second embodiment.

On the other hand, as shown in FIG. 22(c), in the 8-neighbour expansion process, it can be assumed that a part of the human body is shown in the upper right and upper left images. Updates can be suppressed.

Therefore, it can be seen that the 8-neighbor expansion process is effective, but the 4-neighbor expansion process may be adopted depending on the shape of the moving object.

FIG. 23 is an example of a flow chart showing how the extension processing unit 57 performs extension processing on the divided image. In the processing of FIG. 23, mainly the differences from FIG. 19 will be explained.

Steps S23-1 to S23-4 are added to the process of FIG. At step S23-1, the extension processing unit 57 determines one image after division (step S23-1). Any method may be used to determine images after division.

Next, it is determined whether or not a moving object is detected in the divided image of interest (step S23-2).

If the determination in step S23-2 is Yes, the extension processing unit 57 considers that a moving object has been detected in the up/down/left/right and oblique images (step S23-3). If the determination in step S23-2 is No, the extension processing section 57 does not perform extension processing.

The extension processing unit 57 determines whether or not all divided images are finished (step S23-4), and if finished, synthesizes the divided images (S19-9).

It should be noted that, in step S23-2, it is not judged as Yes for the divided image in which the moving object is considered to be detected. As a result, it is possible to prevent a divided image in which a moving object has been detected and an adjacent divided image from being regarded as having detected a moving object in a chain reaction.

According to the present embodiment, by expanding the divided image for which it is determined that the output image is not to be updated to the vicinity, the person 82 is partly shown, but the similarity to the past input image is equal to or greater than the threshold value y. Even if this happens, the output image is not updated, so it is possible to suppress updating the output image with an input image in which only a part of the moving object is captured.

In the present embodiment, an image processing system 200 that mechanically learns an image containing a human body, builds a learning model 58, and determines whether or not a person is shown in an input image based on the learning model 58 will be described.

FIG. 24 is a functional block diagram of the imaging device 1, communication terminal 3, image management device 5, and terminal device 7 included in the image processing system 200 of this embodiment. 24. In the description of FIG. 24, since the components denoted by the same reference numerals as in FIG. 9 perform the same functions, only the main components of this embodiment may be mainly described.

The image management device 5 of this embodiment has a learning model 58 . The learning model 58 holds learning results of images with or without a moving object, and outputs the presence/absence of a moving object in an input image. The learning model 58 outputs whether or not a moving object appears in the input image (or accuracy, probability, certainty, etc.). You may use the feature-value extracted from the input image. The learning device builds a learning model 58 by learning in advance whether or not a person appears in an input image for training using an algorithm such as deep learning, support vector machine, neural network, or random forest. This learning result is held in the learning model 58, and functions as a judgment unit that judges whether or not the input image contains a moving object (whether or not the image is a subject).

<About judgment using learning model>
1. Discarding Input Image Using Learning Model Only A method of determining an output image using the learning model 58 will be described with reference to FIG. FIG. 25(a) shows an input image, and FIG. 25(b) shows an output image. The determination unit 55 uses the learning model 58 alone to determine an output image based on whether or not a moving object is captured.

Time t ₀ : When the input image 86 — t ₀ is captured, it is input to the learning model 58 . The learning model 58 determines that no person is shown. Since there is no output image at time _t0 , the determination unit 55 determines to output the input image _{86_t0} .

Time t ₁ : When the input image 86 — t ₁ is captured, it is input to the learning model 58 . Since the learning model 58 determines that _no person is captured, the input image _{86_t1} at time t1 is determined as the output image _{87_t1} .

Time t2: Since the person 82 is shown in the input image _{86_t2} , the learning model ₅₈ determines that the person is shown, discards the input image _{86_t2} , and uses the output image _{87_t1} at time _t1 as the output image. decide.

Time t ₃ to t ₅ : The learning model 58 determines that the person is still photographed because the person 82 is still photographed. For this reason, the output images 87 — t ₂ to 87 — t ₄ one period before are determined as the output images as they are.

Time t ₆ : Since the person 82 picked up the product and left the front of the product shelf 81, the product shelf 81 is shown in the input image, but the number of products is one less. Since the learning model 58 determines that no person is captured, the input image _{86_t6} at time _t6 is determined as the output image _{87_t6} .

In this way, by using the learning model 58, it is possible to discard an input image containing a person with the accuracy of the learning model 58. FIG.

FIG. 26 is an example of a flowchart showing a procedure for discarding an input image by the learning model. The processing in FIG. 26 is repeatedly executed each time an input image is captured. First, an input image is divided into blocks (S15-1).

The learning model determines whether or not the input image contains a moving object (S26-1). If the input image does not include a moving object, the learning model updates the output image with the input image (S26-2).

If the input image contains a moving object, the learning model outputs the output image of one period before (S26-3).

Then, the learning model 58 determines whether or not the processing of all divided images has been completed (S15-6). If all divided images have not been processed, the processing from step S26-1 is repeated, and if all divided images have been processed, the divided images are synthesized (S15-7).

2. Discarding Input Image Based on Similarity Between Input Image and Output Image Not Discarded by Learning Model A method of determining an output image using the learning model 58 will be described with reference to FIG. FIG. 27(a) shows an input image, and FIG. 27(b) shows an output image. The determination unit 55 uses the learning model 58 and determines the output image based on whether the similarity between the output image and the input image is equal to or greater than a threshold.

Time t ₀ : When the input image 86 — t ₀ is captured, it is input to the learning model 58 . The learning model 58 determines that no person is shown. Since there is no output image at time _t0 , the determination unit 55 determines to output the input image _{86_t0} .

Time t ₁ : When the input image 86 — t ₁ is captured, it is input to the learning model 58 . The learning model 58 determines that no person is shown. Next, the similarity calculator 53 calculates the similarity between the output image _{87_t0} and the input image _{86_t1} . Since there is no change in the products on the product shelf 81, the similarity between the input image _{86_t1} and the output image _{87_t0} is high and exceeds the threshold. When the degree of similarity is equal to or greater _than the threshold, the determination unit 55 determines the input image _{86_t1} at time t1 as the output image _{87_t1} .

Time t ₂ : Since the person 82 appears in the input image 86_t ₂ , the learning model 58 determines that the person appears. _The determination unit 55 discards the input image _{86_t2} and directly determines the output image _{87_t1} at time t1 as the output image.

Time t ₃ to t ₅ : The learning model 58 determines that the person is still photographed because the person 82 is still photographed. The determination unit 55 determines the output images 87 — t ₂ to 87 — t ₄ of one period before as the output images as they are.

Time t ₆ : Since the person 82 picked up the product and left the front of the product shelf 81, the product shelf 81 is shown in the input image, but the number of products is one less. First, the learning model 58 determines that no person is in the image. The degree of similarity between the input image _{86_t6} at time _t6 and the output image _{87_t5} at time _t5 is moderate. When the degree of similarity is medium (greater than or equal to the threshold), the determination unit 55 determines the input image _{86_t6} at time _t6 as the output image _{87_t6} .

In this way, by using the learning model 58, it is possible to discard an input image containing a person with the accuracy of the learning model 58. FIG. Since the similarity calculation unit 53 calculates the degree of similarity with respect to the input image in which the learning model 58 determines that the person is not shown, the input image in which the person is actually shown can be discarded.

<Operation procedure>
FIG. 28 is an example of a flowchart showing a procedure for the image manager 5 to determine an output image. In the explanation of FIG. 28, mainly the difference from FIG. 15 will be explained.

First, the image dividing unit 52 divides the omnidirectional image (step S15-1).

Next, the learning model 58 determines whether or not a person is shown in the divided image (S26-1). If it is determined that a person is captured, the process proceeds to step S15-5, and the determination unit 55 determines to output the output image of one cycle before as it is (step S15-5).

If it is determined that a person is not captured, the process proceeds to step S26-1, and the learning model 58 determines whether or not a moving object is captured in each divided image (step S26-1). Subsequent processing may be the same as in FIG.

<Summary>
Therefore, according to the image processing system 200 of this embodiment, since the learning model 58 is used, if the accuracy of the learning model 58 is high, an input image including a person can be detected with high accuracy. The images in which the learning model 58 determines that a person is shown do not include the images in which the product shelf has been updated. The image is not updated. When it is desired to update the output image early with the input image when the product shelf has been updated, the output image can be updated early in a situation where the human body is less likely to appear in the image by reducing the threshold value.

Moreover, the processing using the learning model 58 can be applied not only in the first embodiment but also in combination with any of the second and third embodiments. In combination with any of the embodiments, the learning model 58 may first determine whether or not a person appears in the input image. In the third embodiment, when the learning model 58 determines that a person is shown, the expansion processing unit 57 performs expansion processing centering on the block in which the person exists.

Further, in this embodiment, the learning model 58 determines whether or not a person is captured before the similarity between the input image and the output image is calculated. The learning model 58 may determine whether or not a person is shown. Since the degree of similarity is higher than the threshold, even if the determination unit 55 determines to update the output image with the input image, the input image can be discarded if a part of the human body is captured. That is, since it is necessary to update the output image with the input image for updating the product shelf, there is a limit to increasing the threshold value (threshold value x in the second embodiment). is updated, and even if there is an input image containing a human body, the learning model 58 can detect this and discard the input image containing a human body.

In this embodiment, a modified example of the image processing system 200 will be described.

<About the device that removes the moving object>
In Embodiments 1 to 3, the image management device 5 removes the moving object, but the removal of the moving object may be performed by any of the imaging device 1, the communication terminal 3, the wireless router 9a, or the terminal device 7. FIG. Alternatively, an information processing device such as a computer or microserver existing on the store side may perform the processing. Further, although not shown in FIG. 4, an information processing apparatus dedicated to removing a moving object connected via the communication network 9 may remove the moving object.

Also, in FIG. 4, the terminal device 7 acquires the omnidirectional image with the moving object removed from the image management device 5 , but the terminal device 7 may directly acquire the omnidirectional image from the imaging device 1 . In this case also, any of the imaging device 1, the communication terminal 3, the wireless router, or the terminal device 7 may remove the moving object. Alternatively, the imaging device 1 and the terminal device 7 may be directly connected. Also, the terminal device 7 and the imaging device 1 may be configured integrally.

Further, when the terminal device 7 and the imaging device 1 are integrated, the terminal device 7 transmits the omnidirectional image to the image management device 5, and acquires the omnidirectional image from which the moving object is removed from the image management device 5. good too.

<Usage scenes other than stores>
In order to share information displayed on a whiteboard or an electronic blackboard used in meetings, lectures, etc. with a plurality of terminal devices 7, the imaging device 1 may capture an image of the whiteboard or the electronic blackboard. Although the writer may appear in the image, the moving object removal described in this embodiment can remove the writer, and the output image can be updated by detecting changes in the information displayed on the whiteboard or electronic blackboard.

In addition, a live camera can also be suitably used. A live camera is an imaging device 1 that is installed at a remote location to monitor video from a user's location and periodically captures video. For example, the image management device 5 can remove the moving object when it is installed at a weather observation site or a sightseeing spot and an environmental visitor or the like appears in the omnidirectional image. The imaging device 1 may be placed indoors such as in a museum as well as outdoors.

The moving object removal method of the present embodiment can also be applied to inventory management in which the image management device 5 replenishes products by analyzing the image of the imaging device 1 placed in a distribution warehouse or the like. Since workers who pick up and replenish products are working in distribution warehouses, they may appear in the omnidirectional image. A change in the omnidirectional image due to reduction can be output as an output image.

The moving object removal method of the present embodiment can also be applied to abnormality monitoring in which an image of a factory line is captured by the imaging device 1 and an abnormality of the line is detected by the image management device 5 . In the factory line, workers perform assembly, etc., but in this embodiment, workers are removed from the omnidirectional image, and changes in the omnidirectional image due to changes in products on the line are output as output images. can.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

For example, in the present embodiment, an example of fixed-point observation in which 360-degree surroundings (the same effect as imaging in the same direction) is taken from the same place has been described, but the imaging device 1 does not have to be fixed. For example, imaging is possible even when the user is holding the camera. Also, when the imaging device 1 moves slowly, the degree of similarity due to changes in the background is intermediate or higher, so the output image can be updated with the input image.

Also, the omnidirectional images displayed by the terminal device 7 may be stored in chronological order. As a result, it is possible to reproduce the change of only the product shelf as to how the products on the product shelf have changed. It is effective not only for product shelves but also for recording weather changes.

Further, the configuration examples shown in FIGS. 9 and 20 shown in the above embodiment are divided according to main functions in order to facilitate understanding of the processing of the image processing system 200 . However, the method of dividing each processing unit and the names thereof do not limit the present invention. The image processing system 200 can also be divided into more processing units according to the processing content. Also, one processing unit can be divided to include more processing.

Also, the functions of the image management device 5 may be distributed to a plurality of server devices, and the image processing system 200 may have a plurality of image management devices 5 .

Further, the transmission/reception unit 51 is an example of image acquisition means, the similarity calculation unit 53 is an example of similarity calculation means, the determination unit 55 and learning model 58 are examples of determination means, and the determination unit 55 and learning model 58 are examples of determination means. Reference numeral 58 denotes an example of subject image determination means, image division section 52 is an example of image division means, extension processing section 57 is an example of processing means, and display control section 73 and transmission/reception section 51 are an example of output means. . The threshold x is an example of the first threshold, the threshold y is an example of the second threshold, the similarity between the input image and the output image is an example of the first similarity, and the current input image and the past The similarity of the input image is an example of the second similarity.

1 imaging device 3 communication terminal 5 image management device 7 terminal device 81 product shelf 82 person 86 input image 87 output image 200 image processing system

Japanese Patent No. 5493709

Claims (14)

information processing equipment,
an image acquisition means for acquiring an image captured by the imaging device as an input image;
subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
determining means for determining to output the input image when determining that the input image is an image of a subject, and determining to output the externally output image when determining that the input image is not an image of a subject;
output means for outputting the input image or the output image according to the determination result of the determination means;
Similarity calculating means for calculating a first similarity between the input image obtained by the image obtaining means and the output image output to the outside;
function as
When the first degree of similarity is equal to or greater than the first threshold, the subject image determination means determines that the input image is the image of the subject, and determines that the first degree of similarity is less than the first threshold. , determining that the input image is not an image of a subject;
When the first similarity is greater than or equal to the first threshold,
The similarity calculation means calculates a second similarity between the current input image acquired by the image acquisition means and an input image that is older than the current input image,
The determination means determines to output the current input image when the second similarity is equal to or greater than the second threshold, and outputs to the outside when the second similarity is less than the second threshold. A program for determining to output the output image. information processing equipment,
an image acquisition means for acquiring an image captured by the imaging device as an input image;
subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
determining means for determining to output the input image when determining that the input image is an image of a subject, and determining to output the externally output image when determining that the input image is not an image of a subject;
output means for outputting the input image or the output image according to the determination result of the determination means;
A learning model that pre-learns an image with or without a moving object and outputs the presence or absence of a moving object in the image;
function as
When the learning model does not determine that the input image acquired by the image acquiring means includes a moving object, the subject image determining means determines that the input image is an image of a subject, and the input image does not include a moving object. if the learning model determines that the input image is not an image of the subject, and
When the learning model does not determine that the input image acquired by the image acquisition means includes a moving object, the information processing device is configured to detect the input image acquired by the image acquisition means and the output image output to the outside. Functioning as similarity calculation means for calculating the first similarity,
When the first similarity is greater than or equal to the first threshold,
The similarity calculation means calculates a second similarity between the current input image acquired by the image acquisition means and an input image that is older than the current input image,
The determination means determines to output the current input image when the second similarity is equal to or greater than a second threshold, and outputs to the outside when the second similarity is less than the second threshold. A program for determining to output the output image. When the first degree of similarity is less than the first threshold, or when the second degree of similarity is less than the second threshold,
2. determining that a moving object, which is captured in front of said subject when it exists between said imaging device and said imaging device, is captured in said current input image obtained by said image obtaining means. Or the program according to 2. 3. The similarity calculating means calculates the second similarity between the input image a predetermined time ago determined based on the moving speed of the moving body and the current input image acquired by the image acquiring means. 4. The program according to any one of 1 to 3. 5. The program according to any one of claims 1 to 4, wherein said second threshold is greater than said first threshold. causing the information processing device to function as image dividing means for dividing the input image acquired by the image acquiring means into blocks;
The similarity calculating means calculates the first similarity for each block of the input image divided into blocks and the output image,
6. The program according to any one of claims 1 to 5, wherein the determination means determines whether to output the current input image or the output image output to the outside for each block. The image dividing means refers to size information in which information about the size of the block is associated with coordinate information of the image captured by the imaging device, and
7. The program according to claim 6, wherein said input image obtained by said image obtaining means is divided into blocks. 3. The size of said block is larger than a subject imaged by said imaging device and smaller than a moving object that is captured in front of said subject when said subject is present between said imaging device and said imaging device. 8. The program according to 6 or 7. 6. The information processing apparatus further functions as processing means for judging that the output image is to be outputted to the outside for each block adjacent to the block judged by the judging means to be outputted to the outside. 9. The program according to any one of 8. 10. The program according to claim 9, wherein the processing means determines to output the output image to the outside for each of the blocks that are eight adjacent to the block determined by the determination means to be output to the outside. an image acquisition means for acquiring an image captured by the imaging device as an input image;
subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
determining means for determining to output the input image when determining that the input image is an image of a subject, and determining to output the externally output image when determining that the input image is not an image of a subject;
output means for outputting the input image or the output image according to the determination result of the determination means;
a similarity calculating means for calculating a first similarity between the input image obtained by the image obtaining means and the output image output to the outside;
When the first degree of similarity is equal to or greater than the first threshold, the subject image determination means determines that the input image is the image of the subject, and determines that the first degree of similarity is less than the first threshold. , determining that the input image is not an image of a subject;
When the first similarity is greater than or equal to the first threshold,
The similarity calculation means calculates a second similarity between the current input image acquired by the image acquisition means and an input image that is older than the current input image,
The determination means determines to output the current input image when the second similarity is equal to or greater than the second threshold, and outputs to the outside when the second similarity is less than the second threshold. and determining to output the output image. an image acquisition means for acquiring an image captured by the imaging device as an input image;
subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
determining means for determining to output the input image when determining that the input image is an image of a subject, and determining to output the externally output image when determining that the input image is not an image of a subject;
output means for outputting the input image or the output image according to the determination result of the determination means;
a learning model function for pre-learning an image with or without a moving object and outputting the presence or absence of a moving object in the image;
When the learning model function does not determine that the input image acquired by the image acquiring means includes a moving object, the subject image determining means determines that the input image is an image of a subject, and the input image does not include a moving object. when the learning model function determines that the input image is not an image of the subject, and
a first similarity between the input image acquired by the image acquisition means and an output image output to the outside when the learning model function does not determine that the input image acquired by the image acquisition means contains a moving object; Calculate the degree,
When the first similarity is greater than or equal to the first threshold,
calculating a second degree of similarity between the current input image acquired by the image acquisition means and an input image that is older than the current input image;
The determination means determines to output the current input image when the second similarity is equal to or greater than a second threshold, and outputs to the outside when the second similarity is less than the second threshold. An information processing apparatus that determines to output the output image. An image processing system having an imaging device that periodically images the surroundings and one or more information processing devices,
an image acquiring means for acquiring an image captured by the imaging device as an input image;
subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
determining means for determining to output the input image when determining that the input image is an image of a subject, and determining to output the externally output image when determining that the input image is not an image of a subject;
output means for outputting the input image or the output image according to the determination result of the determination means;
a similarity calculating means for calculating a first similarity between the input image obtained by the image obtaining means and the output image output to the outside;
When the first degree of similarity is equal to or greater than the first threshold, the subject image determination means determines that the input image is the image of the subject, and determines that the first degree of similarity is less than the first threshold. , determining that the input image is not an image of a subject;
When the first similarity is greater than or equal to the first threshold,
The similarity calculation means calculates a second similarity between the current input image acquired by the image acquisition means and an input image that is older than the current input image,
The determination means determines to output the current input image when the second similarity is equal to or greater than the second threshold, and outputs to the outside when the second similarity is less than the second threshold. and determining to output the output image. An image processing system having an imaging device that periodically images the surroundings and one or more information processing devices,
an image acquiring means for acquiring an image captured by the imaging device as an input image;
subject image determination means for determining whether the input image acquired by the image acquisition means is an image of a subject;
determining means for determining to output the input image when determining that the input image is an image of a subject, and determining to output the externally output image when determining that the input image is not an image of a subject;
output means for outputting the input image or the output image according to the determination result of the determination means;
a learning model function for pre-learning an image with or without a moving object and outputting the presence or absence of a moving object in the image;
When the learning model function does not determine that the input image acquired by the image acquiring means includes a moving object, the subject image determining means determines that the input image is an image of a subject, and the input image does not include a moving object. when the learning model function determines that the input image is not an image of the subject, and
When the learning model function does not determine that the input image acquired by the image acquisition means includes a moving object, the information processing device is configured to perform the input image acquired by the image acquisition means and the output image output to the outside. function as similarity calculation means for calculating the first similarity of
When the first similarity is greater than or equal to the first threshold,
The similarity calculation means calculates a second similarity between the current input image acquired by the image acquisition means and an input image that is older than the current input image,
The determination means determines to output the current input image when the second similarity is equal to or greater than a second threshold, and outputs to the outside when the second similarity is less than the second threshold. An image processing system that determines to output the output image.

Images