JP2017158169A - Image display system, display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display system which facilitates evaluating the advance motion of a moving body.SOLUTION: The image display system comprising one or more information processing device and a display device comprises: acquisition means for acquiring image data obtained by imaging with a wide-angle imaging apparatus capable of imaging the surrounding in a horizontal direction; and display control means which causes the display device to display a moving body reflected in the image data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image display system, a display device, and a program.

  It is known that walking by people helps to promote health. In addition, when walking, it is preferable for health to walk with a correct posture, and it is difficult to fall. Since it is difficult for the person to evaluate the posture at the time of walking, there are cases in which an instructor or the like related to walking objectively analyzes walking motion or tries to guide the analysis result to the pedestrian.

  A motion capture system is known as a system for measuring the posture and motion of a person or the like. In a motion capture system, a camera can image a person with a marker attached to the joint position of the body, etc., and track the movement of the marker with a tracker, so that the computer can analyze the walking motion of a person .

  However, the motion capture system requires a large number of markers in order to capture the movement of a person or the like from various angles, and there is a tendency that it takes time and cost to prepare for imaging of walking motion and to analyze after imaging. In addition, there is a disadvantage that walking motion can be measured only in a special environment such as a laboratory.

  For such inconvenience, a technique has been devised that can analyze a walking motion with a relatively simple configuration using a captured image of a pedestrian with a plurality of marks attached to a predetermined part of the body (for example, Patent Documents). 1).

  However, the technique disclosed in Patent Document 1 has a problem that walking motions that can be imaged are limited. This is because the position and orientation of the camera are fixed while people and the like move gradually by walking. When the camera is fixed, for example, only a walking motion of a pedestrian who walks linearly can be captured, and it is difficult to capture a walking motion of a pedestrian other than a linear walk.

  If there is a system that changes the direction and position of the camera according to the person's walking, the walking motion that can be imaged can be expanded, but considering the cost and space for that, it is easy to realize as a widely used system Not.

  In view of the above problems, an object of the present invention is to provide an image display system that can evaluate the moving motion of a moving object.

  In view of the above problems, the present invention is an image display system having one or more information processing devices and a display device, and acquiring means for acquiring image data captured by a wide-angle imaging device capable of capturing an image in the horizontal direction. And display control means for causing the display device to display a moving object shown in the image data.

  It is possible to provide an image display system that can easily evaluate the moving motion of a moving object.

It is an example of the figure explaining schematic operation | movement of an image display system. 1 is an example of a schematic configuration diagram of an image display system. It is an example of the hardware block diagram of a server. It is an example of the hardware block diagram of an acceleration measuring device. It is an example of the hardware block diagram of an omnidirectional image pick-up device. It is an example of a functional block diagram of a server, a display device, an omnidirectional image capturing device, and an acceleration measuring device. It is an example of the figure explaining how a pedestrian walks the circumference | surroundings of an omnidirectional image imaging device. It is an example of the sequence diagram explaining the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is a figure which shows an example of the image data and measurement data which the display apparatus displayed on the display. It is an example of the flowchart figure which shows the procedure in which an image rotation part specifies the range which the pedestrian is reflected, and displays it on an image data column. It is a figure which shows typically the position of the foot of a pedestrian in the top view of an omnidirectional image. It is an example of the flowchart figure which shows the procedure in which an image rotation part specifies the range which the pedestrian is reflected, and displays it on an image data column. It is an example of the figure which illustrates face recognition typically. It is an example of the figure which illustrates the detection of a moving body typically. It is an example of the figure explaining the display range trimmed from a spherical image. It is an example of the figure explaining the method of determining the center O of a circumscribed rectangle using two omnidirectional images. It is an example of the flowchart figure of the procedure in which a display apparatus always monitors two spherical images and trims a display range. It is an example of the figure explaining the method of switching an omnidirectional image when a pedestrian approaches the edge part. It is an example of the flowchart figure of the procedure which switches a celestial sphere image and a display apparatus trims a display range when a pedestrian approaches the edge part. It is an example of the figure explaining the center O of a circumscribed rectangle in case two pedestrians are imaged. It is an example of the figure explaining the center of the circumscribed rectangle when a pedestrian overlaps. It is an example of the flowchart figure of the procedure in which a display apparatus specifies the position of several pedestrians and trims a display range. (Example 2) which is an example of the functional block diagram of a display apparatus. It is an example of the figure explaining the pedestrian who turns. It is a figure which shows an example of the data display screen which the display apparatus displayed on the display. It is an example of the figure explaining inversion of image data. It is an example of the flowchart figure which shows the procedure in which the display apparatus reverses right and left the range where the pedestrian is reflected, and displays it in an image data column. It is an example of the figure explaining the position of a pedestrian. It is a figure explaining the data display screen on which the position of the pedestrian was displayed. It is an example of the figure explaining the determination method of the position of the position icon reversed. It is a figure which shows an example of the data display screen where the position icon was reversed. It is an example of the flowchart figure explaining the procedure in which a display control part displays a data display screen. It is an example of the figure which shows the line containing a linear part and a turning part. It is an example of the figure explaining the data display screen on which the image data during walking on the line containing a straight part and a turning part was displayed. It is an example of the flowchart figure which shows the procedure in which a display control part displays image data. It is another example of the schematic block diagram of an image display system. FIG. 41 is a sequence diagram in which the acceleration measuring device displays measurement data and image data in the configuration as shown in FIG. It is an example of a functional block diagram of a server, a display device, an omnidirectional image capturing device, and an acceleration measuring device.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Outline of image processing system>
FIG. 1 is an example of a diagram illustrating a schematic operation of the image display system 100 according to the present embodiment. An omnidirectional image capturing device 60 is installed, and the omnidirectional image capturing device 60 captures a pedestrian walking around the omnidirectional image capturing device 60. The pedestrian wears the acceleration measuring device 50 on the body, and the acceleration measuring device 50 measures the acceleration generated in the pedestrian being imaged.

  (1) The omnidirectional image capturing device 60 transmits image data to the server 10, and the acceleration measuring device 50 transmits acceleration measurement data to the server 10.

  (2) The imaging start time of image data and the measurement start time of measurement data are synchronized, and the imaging end time of image data and the measurement end time of measurement data are synchronized. For this reason, the server 10 can synchronize the time axes of image data and measurement data.

  (3) The server 10 transmits the image data and measurement data whose time axes are synchronized to the display device 30. Thereby, the display device 30 can display the image data and the measurement data on the same time axis.

  (4) The display device 30 detects a pedestrian from the omnidirectional image, trims a display range including the pedestrian, and displays it on the display. Since the omnidirectional image can pick up 360 degrees around the omnidirectional image, the pedestrian can take an image with the same size regardless of the orientation of the omnidirectional image pickup device 60. However, if the display device 30 converts the omnidirectional image into a rectangle by the Mercator projection or the like, the image may be distorted and it may be difficult for a leader or the like to see. Moreover, a pedestrian's image will also become small. Therefore, the display device 30 can display a pedestrian with less distortion on the display by performing trimming as described above.

  According to such a configuration, first, the omnidirectional image capturing device 60 is used, and a pedestrian walking around the pedestrian is imaged. Therefore, not only a pedestrian moving linearly but also turning or turning left and right. Or pedestrians can be imaged. It is known that a pedestrian tends to lose its center of gravity when turning or turning left and right, and being able to image a pedestrian making a turn or turning left and right is effective for analysis of walking motion. Further, since the omnidirectional image capturing device 60 can always capture the side surface of the pedestrian without moving the camera, it can capture how the pedestrian's body shakes in the front-rear direction. In addition, since the pedestrian is always displayed large in an image with little distortion, the video is easy to see for the instructor. An instructor or the like who walks can check the walking motion of the pedestrian with a video, evaluate the walking motion, and guide improvement points.

  In addition, since the measurement data that measures the acceleration of the pedestrian is displayed on the same time axis as the image data, if the instructor finds a walking motion that should be improved, the measured data indicates that the walking motion is not good. Can be supported by On the other hand, by comparing the measurement data prepared in advance, which is considered good, with the measurement data of the pedestrian (subject), the instructor can find undesirable measurement data. Since the unfavorable measurement data is associated with the image data, the instructor or the like can specify the walking motion to be improved from the image data. Therefore, the walking motion can be evaluated by using the measurement data and the image data in a complementary manner.

<Terminology>
A moving body refers to an object that travels forward, backward, laterally, and the like. Alternatively, it is an object that travels with the movement of the center of gravity. Specific examples include humans, animals, robots, and the like, but in the following description, humans (pedestrians) will be described as an example for explanation. Walking includes running. You may also walk backwards. In addition, wheelchairs on which people have boarded, people wearing artificial legs, bicycles, stilts, tricycles, etc. are also moving objects.

  The moving object walks while changing its direction or changing its direction. “While changing the direction” means that at least the walking line is not a straight line. “Changing the direction” means that two or more straight lines can be found from the walking line. Specifically, the moving body turns and turns right and left. A line refers to a path, path, or locus along which a moving object moves.

  Further, acceleration is an example of information that can evaluate the traveling motion of a traveling object. The information that can evaluate the progress of the progressing object refers to information that varies with the progress of the progressing object. Alternatively, it may be information that varies with progress when attached to a target that travels. Or you may say that progress operation | movement is the information which can evaluate quality etc. Specifically, it is acceleration, angular velocity, position or orientation.

  Correspondence between image data and measurement data means that when one or more pieces of image data are determined from some image data, one or more pieces of measurement data are determined. Alternatively, image data and measurement data measured at substantially the same time are specified, and when one of time, image data, or measurement data is determined, the rest is determined.

  A leader refers to a person who can confirm a walking motion of a pedestrian with a video, evaluate the walking motion, and plan improvement points. Specific examples include holders of qualifications such as public health nurses, clinical laboratory technicians, and prosthetic limb orthoses. Moreover, even if it is not a person who can evaluate a walking motion and can plan an improvement point etc., operation of the image display system 100 is possible, and a leader may be called an operator, a user, or a user.

<System configuration example>
FIG. 2 shows an example of a schematic configuration diagram of the image display system 100 of the present embodiment. The image display system 100 includes a server 10, an acceleration measuring device 50, an omnidirectional image capturing device 60, and a display device 30 that can communicate with each other via a network N.

  The network N is constructed by a LAN constructed in a facility where the image display system 100 is arranged, a provider network of a provider that connects the LAN to the Internet, a line provided by a circuit provider, and the like. When the network N has a plurality of LANs, the network N is called a WAN or the Internet. The network N may be constructed by either wired or wireless, and wired and wireless may be combined. When the display device 30 is directly connected to a line telephone network or a mobile phone network, it can be connected to a provider network without going through a LAN.

  The omnidirectional image capturing device 60 is an image capturing device (camera) that captures a 360 ° surrounding landscape called an omnidirectional image or an omnidirectional image (hereinafter simply referred to as an omnidirectional image or an omnidirectional image). By providing an optical system to be described later, the omnidirectional image capturing device 60 can obtain an omnidirectional image by one imaging operation. Since the omnidirectional image has an image of 360 ° around the blind spot, the instructor can selectively display an arbitrary area on the display or the like for viewing. In the present embodiment, it is not necessary to have an imaging range of 360 degrees in the latitude direction (up and down direction of the image). This is because pedestrians are not shown in the ceiling and floor areas. Further, it may not be possible to capture an image in the entire range of 360 degrees in the longitude direction (the horizontal direction of the image). This is because it may not be necessary to turn the entire omnidirectional image capturing device 60 in order for the instructor to analyze the walking motion of the pedestrian. Further, as the omnidirectional image capturing device 60, an image capturing device that obtains an omnidirectional image by connecting a plurality of images captured by an image capturing device that captures a rectangular range of a certain angle of view may be used.

  The acceleration measuring device 50 is a device that measures the acceleration of a pedestrian. The acceleration is preferably measured on each of the three axes of vertical acceleration, traveling acceleration, and lateral acceleration, but may be acceleration in any one direction. Although there is a dedicated sensor for measuring acceleration as the acceleration measuring device 50, the acceleration measuring device 50 may be a device having an acceleration sensor. Specific examples include smartphones, mobile phones, tablet terminals, notebook PCs, wearable PCs (mounted as head mounted displays, sunglasses, watches, etc.), pedometers, and the like, but are not limited thereto.

  Further, the acceleration measuring device 50 may have a function of measuring angular velocity, position, orientation, etc. in addition to acceleration. The sensor that measures the angular velocity is called a gyro sensor, and the position is provided to the acceleration measuring device 50 by a GPS satellite, IMES, or a beacon. A sensor that detects the direction of east, west, north, and south, for example, with reference to the north is called an orientation sensor. Any of these can be used for the evaluation of the walking motion as well as the acceleration. In the present embodiment, the acceleration measuring device 50 is described as measuring acceleration for the sake of simplicity, but the acceleration measuring device 50 can transmit measurement data used for evaluating these walking motions to the server 10.

  The server 10 is an information processing device that receives and processes image data from the omnidirectional image capturing device 60, or receives measurement data and synchronizes the image data with the time axis. The function of the server 10 can be shared by the acceleration measuring device 50 and the display device 30.

  The display device 30 is an information processing device used as a terminal device that allows an instructor or the like to display image data and measurement data. Specifically, for example, PC (Personal Computer), smart phone, tablet device, mobile phone, car navigation terminal, wearable computer (head mounted display, sunglasses, wristwatch, etc.), camera, electronic blackboard, projector, game machine, or MFP (Multifunction Peripheral / Printer / Product) etc. are mentioned.

<Hardware configuration example>
FIG. 3 is an example of a hardware configuration diagram of the server 10. Note that the hardware configuration of the server 10 shown in the figure does not need to be housed in a single casing or provided as a single device, and represents a hardware element that is preferably provided in the server 10. . The configuration of the server 10 may be determined by cloud computing in which resources are appropriately allocated according to a load or the like.

  The server 10 includes a CPU 101, ROM 102, RAM 103, HDD 105, media drive 107, display 108, network I / F 109, keyboard 111, mouse 112, and optical drive 114 connected to the bus 110. The CPU 101 executes an image processing program stored in the HD 104 and controls the overall operation of the server 10. The ROM 102 stores a program used for driving the CPU 101 such as an IPL (Initial Program Reader). A RAM 103 is a main storage device used as a work area for the CPU 101. The HD 104 is a storage device equipped with a nonvolatile memory.

  An HDD (Hard Disk Drive) 105 controls reading or writing of various data with respect to the HD 104 according to the control of the CPU 101. A display (display device 30) 108 displays various information such as a cursor, menu, window, character, or image. A network I / F 109 is an interface with the network N.

  A keyboard 111 and a mouse 112 are input / output devices. The keyboard 111 includes a plurality of keys for inputting characters, numerical values, various instructions, and the like, and receives inputs from these keys. The mouse 112 receives movement of the mouse pointer, selection and execution of various instructions, selection of a processing target, and the like.

  The media drive 107 controls reading or writing (storage) of data with respect to the recording medium 106 such as a flash memory. The optical drive 114 controls reading or writing of various data with respect to a CD (Compact Disc) 113 as an example of a removable recording medium.

  The image processing program may be installed in an installable or executable format, recorded on a computer-readable recording medium such as the recording medium 106 or the CD 113, and distributed. Alternatively, the image processing program may be distributed in a form downloaded from any server 10 type information processing apparatus.

  It should be noted that the hardware configuration of the display device 30 is the same as or different from that in FIG.

  FIG. 4 shows an example of a hardware configuration diagram of the acceleration measuring device 50. In FIG. 4, a smartphone or a tablet device is assumed as the acceleration measuring device 50, but the hardware configuration of the acceleration measuring device 50 is not limited to this.

  The acceleration measuring apparatus 50 includes a CPU 201, a ROM 202, a RAM 203, a flash memory 204, a CMOS sensor 205, an acceleration / orientation sensor 206, a gyro sensor 207, a media drive 208, an audio input unit 211, and an audio output unit that are connected to the bus 210. 212, a communication device 213, a GPS receiver, a display 215, and a touch panel 216. The bus 210 is an address bus, a data bus, or the like for electrically connecting these units.

  The CPU 201 controls the overall operation of the display device 30 by executing a program stored in the flash memory 204. The ROM 202 stores IPL and basic input / output programs. A RAM 203 is a main storage device used as a work area when the CPU 201 executes a program. The flash memory 204 is a nonvolatile storage device that stores programs executed by the display device 30 and various data. The program is, for example, an operating system and a device program executed by the omnidirectional image capturing device 60.

  The CMOS sensor 205 is an image sensor that captures an image of a subject under the control of the CPU 201 and obtains image data. A CCD sensor may be used instead of the CMOS sensor 205. The acceleration / orientation sensor 206 has a function as an electronic magnetic compass for detecting geomagnetism, a function for measuring acceleration in three axial directions, and the like. The gyro sensor 207 detects an angular velocity when the acceleration measuring device 50 rotates with respect to the x axis, the y axis, or the z axis. The rotation angle of each axis is called a yaw angle, a pitch angle, and a roll angle.

  The media drive 208 controls reading or writing (storage) of data with respect to a recording medium 209 such as a flash memory. According to the control of the media drive 208, the recording medium 209 from which already recorded data is read or newly written and stored is detachable.

  The voice input unit 211 is a microphone that converts voice into a voice signal. The audio output unit 212 is a speaker that converts an audio signal into audio. The communication device 213 communicates with the nearest wireless station device 9 by a wireless communication signal using the antenna 213a. Alternatively, the communication device 213 may be a LAN card connected to the LAN. The GPS receiving unit 214 detects position information (latitude, longitude, and altitude) of the display device 30 by an IMES (Indoor MEssaging System) as a GPS satellite or indoor GPS.

  The display 215 displays an omnidirectional image under the control of the CPU 201 and displays various menus, icons, and the like for the user to operate the display device 30. The touch panel 216 is integrally superimposed on the display 215 and detects a touch position (coordinates) on the display 215 with respect to a touch with a finger or a touch pen.

  The device program may be recorded in a computer-readable recording medium such as the recording medium 209 and distributed as a file in an installable or executable format. Alternatively, the device program may be distributed in a form downloaded from any server 10 type information processing device.

  FIG. 5 is an example of a hardware configuration diagram of the omnidirectional image capturing apparatus 60. In the following, the omnidirectional image capturing device 60 is an omnidirectional image capturing device using two image capturing elements, but the number of image capturing elements may be three or more. In addition, it is not always necessary to use an apparatus dedicated to omnidirectional imaging. By attaching a retrofit omnidirectional imaging unit to a normal digital camera or smartphone, it has substantially the same function as the omnidirectional imaging apparatus 60. It may be.

  As shown in FIG. 5, the omnidirectional image capturing apparatus 60 includes an image capturing unit 301, an image processing unit 304, an image capturing control unit 305, a microphone 308, a sound processing unit 309, an audio output unit 319, a CPU 311, a ROM 312, An SRAM 313, a DRAM 314, an operation unit 315, a network I / F 316, a communication device 317, and an antenna 317a are provided.

  Among these, the imaging unit 301 includes wide-angle lenses (so-called fish-eye lenses) 302a and 302b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 303a and 303b are provided. The image pickup elements 303a and 303b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by a fisheye lens into image data of an electrical signal and output the image data. A timing generation circuit for generating a vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 303a and 303b of the imaging unit 301 are each connected to the image processing unit 304 by a parallel I / F bus. On the other hand, the imaging elements 303 a and 303 b of the imaging unit 301 are connected to a serial I / F bus (I2C bus or the like) separately from the imaging control unit 305. The image processing unit 304 and the imaging control unit 305 are connected to the CPU 311 via the bus 310. Further, ROM 312, SRAM 313, DRAM 314, operation unit 315, network I / F 316, communication device 317, electronic compass 318, and the like are also connected to the bus 310.

  The image processing unit 304 takes in the image data output from the image sensors 303a and 303b through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. Creates equirectangular image data.

  The imaging control unit 305 generally sets commands and the like in the register groups of the imaging elements 303a and 303b using the I2C bus with the imaging control unit 305 as a master device and the imaging elements 303a and 303b as slave devices. Necessary commands and the like are received from the CPU 311. The imaging control unit 305 also takes in the status data of the register groups of the imaging elements 303a and 303b using the I2C bus and sends them to the CPU 311.

  The imaging control unit 305 instructs the imaging elements 303a and 303b to output image data at the timing when the shutter button of the operation unit 315 is pressed. Some omnidirectional image capturing devices 60 may have a preview display function by a display or a function corresponding to a moving image display. In this case, output of image data from the image sensors 303a and 303b is continuously performed at a predetermined frame rate (frame / minute).

  Further, as will be described later, the imaging control unit 305 also functions as a synchronization control unit that synchronizes the output timing of the image data of the imaging elements 303a and 303b in cooperation with the CPU 311. In the present embodiment, the omnidirectional image capturing device 60 is not provided with a display unit, but may be provided with a display unit.

  The microphone 308 converts sound into sound (signal) data. The sound processing unit 309 takes in sound data output from the microphone 308 through the I / F bus and performs predetermined processing on the sound data. The audio output unit 319 is a speaker that converts an audio signal into audio.

  The CPU 311 controls the overall operation of the omnidirectional image capturing apparatus 60 and executes necessary processes. The ROM 312 stores various programs for the CPU 311. The SRAM 313 and the DRAM 314 are work memories, and store programs executed by the CPU 311 and data being processed. In particular, the DRAM 314 stores image data being processed by the image processing unit 304 and image data created by a processed equirectangular projection.

  The operation unit 315 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user inputs various imaging modes, imaging conditions, and the like by operating the operation buttons.

  The network I / F 316 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. The network I / F 316 may be a network interface regardless of wireless or wired. Data of the image created by the equirectangular projection stored in the DRAM 314 is recorded on an external medium via the network I / F 316, or the network I / F 316 which becomes a network I / F as necessary. Or transmitted to an external device.

  The communication device 317 communicates with an external device via a wireless technology such as WiFi (wireless fidelity), NFC, or LTE (Long Term Evolution) via an antenna 317a provided in the omnidirectional image capturing device 60. The communication device 317 can also transmit image data created by equirectangular projection to an external device.

  The electronic compass 318 calculates the azimuth and tilt (Roll rotation angle) of the omnidirectional imaging device 60 from the earth's magnetism, and outputs azimuth / tilt information. This azimuth / tilt information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. The related information includes each data of the image capturing date and time and the data capacity of the image data.

<Functional configuration>
Next, functional configurations of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 will be described with reference to FIG. FIG. 6 is an example of a functional block diagram of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

<< Each functional configuration of server 10 >>
The server 10 includes a communication unit 11, an image adjustment unit 12, an acceleration data processing unit 13, an operation input receiving unit 14, a synchronization instruction unit 15, and a storage / reading unit 19. Each of these units included in the server 10 is realized by any one of the components illustrated in FIG. 3 operating according to an instruction from the CPU 101 according to the image processing program 1010 expanded from the HD 104 onto the RAM 103. Function or means to be functioned.

  In addition, the server 10 includes a storage unit 1000 constructed by one or more of the HD 104, the ROM 102, and the RAM 103 illustrated in FIG. The storage unit 1000 stores an image processing program 1010, a measurement data DB 1001, and an image data DB 1002. The measurement data DB 1001 is a storage unit that stores measurement data transmitted from the acceleration measurement device 50. The image data DB 1002 is a storage unit that stores image data transmitted from the omnidirectional image capturing device 60.

  The communication unit 11 of the server 10 is realized by a command from the CPU 101 illustrated in FIG. 3 and a network I / F 109, and the like, and via the network N, the acceleration measuring device 50, the omnidirectional image capturing device 60, and Various data are transmitted to and received from the display device 30. In the following description, even when the server 10 communicates with the function of the communication unit 11, the description “communication via the communication unit 11” may be omitted. The same applies to the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

  The image adjustment unit 12 is realized by a command or the like from the CPU 101 shown in FIG. 3, and synchronizes the time axis of the image data acquired from the omnidirectional image capturing device 60 and the measurement data acquired from the acceleration measuring device 50. .

  The acceleration data processing unit 13 is realized by a command or the like from the CPU 101 shown in FIG. 3, and converts discrete measurement data into a graph with respect to the time axis by smoothly combining them.

  The operation input receiving unit 14 is realized by a command from the CPU 101 illustrated in FIG. 3, a keyboard 111, a mouse 112, and the like, and receives various operations and inputs to the display device 30 by a leader or the like.

  The synchronization instructing unit 15 is realized by an instruction from the CPU 101 shown in FIG. 3 and performs processing related to synchronization of measurement data and image data. Specifically, the omnidirectional image capturing device 60 and the acceleration measuring device 50 are requested to start measurement and start imaging at the same time, or the omnidirectional image capturing device 60 and the acceleration measuring device to end measurement and image capturing at the same time. Or request to 50. Also, the absolute time is provided to the omnidirectional image capturing device 60 and the acceleration measuring device 50.

  The storage / reading unit 19 is realized by an instruction from the CPU 101 and the HDD 105 shown in FIG. 3, and stores various data in the storage unit 1000 or reads various data stored in the storage unit 1000. I do. In the following description, even when the server 10 accesses the storage unit 1000, the description “read / write via the storage / reading unit 19” may be omitted. The same applies to the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

<< Functional configuration of display device 30 >>
The display device 30 includes a communication unit 31, a display control unit 32, an operation input reception unit 33, an image rotation unit 34, a moving object detection unit 35, and a storage / readout unit 39.

  Each unit included in the display device 30 is realized by any one of the constituent elements illustrated in FIG. 3 operating according to a command from the CPU 101 according to the terminal program 3010 expanded from the HD 104 onto the RAM 103. A function, or a means to be functioned. In addition, the function implement | achieved by the web page (The program described by HTML data, XML data, JavaScript (trademark), etc.) which is not restricted to the terminal program 3010, and a function is included. .

  In addition, the display device 30 includes a storage unit 3000 configured by one or more of the HD 104, the ROM 102, and the RAM 103 illustrated in FIG. The storage unit 3000 stores a terminal program 3010. The storage unit 3000 stores measurement data and image data transmitted from the server 10.

(Function of the display device 30)
The communication unit 31 of the display device 30 is realized by a command from the CPU 101 illustrated in FIG. 3 and a network I / F 109 and the like, and transmits and receives various data to and from the server 10 via the network N.

  The display control unit 32 is realized by a command from the CPU 101 shown in FIG. 3 and the like, and displays the image data and measurement data transmitted from the server 10 on the display 108.

  The operation input accepting unit 33 is realized by a command from the CPU 101 shown in FIG. 3, a keyboard 111, a mouse 112, and the like, and accepts various operations and inputs to the display device 30 by an instructor or the like.

  The moving object detection unit 35 is realized by an instruction from the CPU 101 shown in FIG. 3 and detects a moving object from the omnidirectional image. This moving body is mainly a pedestrian. Since the pedestrian is detected, the image rotation unit 34 can trim the image including the pedestrian.

  The image rotation unit 34 is realized by a command or the like from the CPU 101 shown in FIG. 3, and the display range is determined from image data that is an omnidirectional image so that an area where a pedestrian is shown is displayed on the display 108. Trim.

  The storage / reading unit 39 is realized by an instruction from the CPU 101 and the HDD 105 shown in FIG. 3, and stores various data in the storage unit 3000 or reads out various data stored in the storage unit 3000. I do.

<< Functional configuration of acceleration measuring device 50 >>
The acceleration measuring device 50 includes a communication unit 51, an acceleration detection unit 52, and a storage / readout unit 59. These components of the acceleration measuring device 50 are realized by any one of the components shown in FIG. 4 operating according to commands from the CPU 201 according to the device program expanded from the flash memory 204 onto the RAM 203. Function or means to be functioned.

  Further, the acceleration measuring apparatus 50 includes a storage unit 5000 constructed by any one or more of the flash memory 204, the ROM 202, and the RAM 203 shown in FIG. The storage unit 5000 stores an apparatus program 5010. The storage unit 5000 stores measurement data such as acceleration measured by the acceleration measuring device 50.

(Function of acceleration measuring device 50)
The communication unit 51 of the acceleration measuring device 50 is realized by a command from the CPU 201 illustrated in FIG. 4 and the communication device 213 and the like, and transmits and receives various data to and from the server 10 via the network N.

  The acceleration detection unit 52 is realized by a command from the CPU 201 shown in FIG. 4, the acceleration / direction sensor 206, and the like, and measures the acceleration generated in the acceleration measuring device 50. Acceleration is measured at regular intervals or irregularly to obtain acceleration that varies with time. Measurement data including time-series acceleration is stored in the storage unit 5000.

  The storage / reading unit 59 is realized by the instruction from the CPU 201 illustrated in FIG. 4, the flash memory 204, the ROM 202, the RAM 203, and the like, and stores various data in the storage unit 5000 or stored in the storage unit 5000. Performs processing to read various data.

<< Functional configuration of omnidirectional imaging device 60 >>
The omnidirectional image capturing device 60 includes a communication unit 61, an image capturing unit 62, and a storage / reading unit 69. Each of these components included in the omnidirectional image capturing device 60 operates according to an instruction from the CPU 311 according to the image capturing program developed from the ROM 312 to the SRAM 313 or the DRAM 314. A function realized by or a means to be functioned.

  Further, the omnidirectional image capturing device 60 includes a storage unit 6000 constructed by one or more of the ROM 312, the SRAM 313, and the DRAM 314 shown in FIG. 5. The storage unit 6000 stores image data of an omnidirectional image and an imaging program 6010.

(Function of the omnidirectional imaging device 60)
The communication unit 61 of the omnidirectional image capturing apparatus 60 is realized by a command from the CPU 101 illustrated in FIG. 5 and a network I / F 316 or the like, and transmits and receives various data to and from the server 10 via the network N.

  The imaging unit 62 is realized by a command from the CPU 101 shown in FIG. 5, an imaging unit 301, an image processing unit 304, and the like, and captures an omnidirectional image and generates image data. The spherical image may be a still image or a moving image. In any case, the imaging unit 62 images a walking pedestrian over time. In the case of a moving image, the omnidirectional image is captured at regular intervals or irregularly. In the case of a still image, a plurality of omnidirectional images are captured at time intervals while the pedestrian is walking. Image data including time-series spherical images is stored in the storage unit 6000.

  The storage / reading unit 69 is realized by any one or more of the instruction from the CPU 201 shown in FIG. 5, ROM 312, SRAM 313, and DRAM 314, and stores various data in the storage unit 6000 or storage in the storage unit 6000. A process for reading out various data is performed.

<About walking of pedestrians>
How the pedestrian walks around the omnidirectional image capturing device 60 will be described with reference to FIG. FIG. 7A is an example of a diagram schematically illustrating a pedestrian 200 walking around the omnidirectional image capturing device 60. The pedestrian 200 wears the acceleration measuring device 50 on the waist or the like. The place where the acceleration measuring device 50 is mounted is preferably a place close to the center of gravity of the pedestrian 200. In addition to the waist, it may be attached to the chest, back, torso, etc. A plurality of acceleration measuring devices 50 may be mounted.

  The pedestrian 200 walks while turning around the omnidirectional image capturing device 60 while changing the direction. A circular line L may be formed on the walking surface in order to facilitate comparison of walking motions between a plurality of pedestrians or to recognize a route on which the pedestrian 200 walks. In addition, markers with numbers or the like that are continuously spaced at equal intervals may be arranged on the line L. When a leader or the like confirms the image data, the position of the pedestrian 200 with respect to the omnidirectional image capturing device 60 can be easily grasped.

  The pedestrian 200 moves forward with the front (chest) facing in the tangential direction of the circle indicated by the line L. Thereby, it walks so that the circumference | surroundings of the omnidirectional image imaging device 60 may turn. While the pedestrian 200 is walking, the acceleration measuring device 50 measures acceleration. The relative positions of the pedestrian 200 and the acceleration measuring device 50 are fixed, and the traveling direction (front) of the pedestrian 200 is the x direction, the left / right direction is the y direction, and the vertical direction is the z direction.

  Note that the pedestrian 200 does not have to walk the entire circular line L, but may walk more than a distance appropriate for the evaluation of the walking motion. Moreover, the pedestrian 200 may walk the circular line L more than 1 round.

  FIG. 7B is an example of a diagram schematically illustrating a pedestrian 200 walking around the omnidirectional image capturing device 60 along a square. In FIG. 7B, for example, a square line L is formed on the walking surface. When the pedestrian 200 walks along the line L (or on the line), the pedestrian 200 changes its direction (to make a right turn), and the omnidirectional image pickup device 60 can take an image of the pedestrian 200 when making a right turn. . Further, if the pedestrian 200 walks in the opposite direction, the omnidirectional image capturing device 60 can capture the pedestrian 200 when turning left.

  The line L has a straight line portion S. Since the pedestrian 200 reciprocates along (or on) the straight line portion S, the omnidirectional image pickup device 60 is a pedestrian 200 walking from behind, a pedestrian 200 performing 180-degree change of direction, And the pedestrian 200 approaching the omnidirectional image pickup device 60 from the front can be picked up.

  Further, the line L on which the pedestrian 200 walks may be an 8-shaped loop or an infinite shape (infinity shape), and is not limited to that shown in Fig. 7. The line L is a walking motion. The shape may be any shape that is preferable for the evaluation.

<Operation procedure>
Subsequently, an operation procedure of the image display system 100 will be described with reference to FIGS. FIG. 8 is an example of a sequence diagram illustrating the overall operation of the image display system 100. FIG. 8 is a sequence diagram when the server 10 instructs the acceleration measuring device 50 and the omnidirectional image capturing device 60 to start measurement and start imaging, respectively.

  S1: When starting measurement of walking motion, a leader or the like inputs an operation for starting measurement of walking motion to the server 10. The operation input receiving unit 14 of the server 10 receives an operation of a leader or the like.

  S2: The communication unit 11 of the server 10 transmits a measurement start request to the acceleration measuring device 50.

  S3: The communication unit 51 of the acceleration measuring device 50 receives the measurement start request. By acquiring the measurement start request, the acceleration measuring device 50 outputs a buzzer sound or the like from the sound output unit 212. Thereby, the pedestrian can grasp | ascertain that a walk may be started. The pedestrian may be informed of the start of walking by a voice message such as “Please start walking”, vibration by the vibration function, display of a message on the display 215, blinking of the display 215, or the like.

  S4: The communication unit 11 of the server 10 transmits an imaging start request to the omnidirectional image imaging device 60.

  S5: The communication unit 61 of the omnidirectional image capturing apparatus 60 receives the imaging start request. Upon acquisition of the imaging start request, the omnidirectional image imaging device 60 outputs a buzzer sound or the like from the audio output unit 319. Thereby, the pedestrian can grasp | ascertain that a walk may be started. Other notification modes are the same as in step S3. In addition, since the omnidirectional image capturing apparatus 60 may enter the pedestrian's field of view, the omnidirectional image capturing apparatus 60 may light a patrol lamp or the like. Thereby, not only a pedestrian but the surrounding leaders can grasp that a pedestrian may start walking. Note that only one of the notifications by the buzzer in steps S3 and S5 is sufficient.

  S6, S7: In response to the measurement start request, the acceleration detector 52 of the acceleration measuring device 50 starts measuring the acceleration. In response to the imaging start request, the imaging unit 62 of the omnidirectional image imaging device 60 starts imaging. The acceleration detector 52 of the acceleration measuring device 50 measures the acceleration that occurs while the pedestrian is walking, and the imaging unit 62 of the omnidirectional image capturing device 60 images the walking pedestrian. In this way, the measurement data measurement start and the image data imaging start can be synchronized almost simultaneously by an instruction from the server 10. Note that the order of steps S2 to S7 is random. The server 10 may transmit the measurement data measurement start and the image data imaging start to the acceleration measurement device 50 and the omnidirectional image imaging device 60 at the same time.

  S8: When the leader or the like determines that the pedestrian has sufficiently walked, the leader or the like inputs an operation to end the measurement of the walking motion to the server 10. The operation input receiving unit 14 of the server 10 receives an operation of a leader or the like.

  S9: The communication unit 11 of the server 10 transmits a measurement end request to the acceleration measuring device 50. The communication unit 11 of the acceleration measuring apparatus 50 receives the measurement end request, and the acceleration detection unit 52 ends the acceleration measurement. Note that a pedestrian or the like may be notified of the end as in step S3.

  S10: The communication unit 11 of the server 10 transmits an imaging end request to the omnidirectional imaging device 60. The communication unit 61 of the omnidirectional image imaging device 60 receives the imaging end request, and the imaging unit 62 ends imaging of the omnidirectional image. Note that a pedestrian or the like may be notified of the end as in step S5. The order of steps S9 and S10 is random. The server 10 may transmit the measurement data measurement end and the image data imaging end to the acceleration measuring device 50 and the omnidirectional image capturing device 60 at the same time.

  In this way, by the instruction from the server 10, the measurement data measurement end and the image data imaging end can be synchronized almost simultaneously.

  S11: The communication unit 51 of the acceleration measuring device 50 transmits the measurement data stored in the storage unit 5000 to the server 10.

  S12: The communication unit 61 of the omnidirectional image capturing apparatus 60 transmits the image data stored in the storage unit 6000 to the server 10.

  S13: The communication unit 11 of the server 10 receives the measurement data and the image data, and stores them in the measurement data DB 1001 and the image data DB 1002, respectively. Then, the acceleration data processing unit 13 of the server 10 processes acceleration data that is discrete data into a line graph or the like. Such processing may be performed by the display device 30.

  S14: Next, the image adjustment unit 12 of the server 10 performs a process of synchronizing the time axes of the measurement data and the image data. Although the measurement data and the image data are acquired at substantially the same time, the number of measurement data and the number of image data (frame number) are often different. For this reason, the image adjustment unit 12 synchronizes the time axis by associating the measurement data with the image data. Since the measurement start time and the imaging start time are almost the same, and the measurement end time and the imaging end time are almost the same, the time axis can be synchronized with the ratio of the number of data of the measurement data and the image data. For example, if the number of measurement data is 400 and the number of image data is 200, two frames of image data are associated with one acceleration of the measurement data. Alternatively, the ratio of the measurement time of each measurement data (acceleration) with respect to the measurement time is calculated, the ratio of the imaging time of each image data (frame) with respect to the imaging time is calculated, and the frame with the closest ratio is associated with the acceleration. Good. Alternatively, the measurement data and the image data may be associated with the elapsed time from the start of measurement and the elapsed time from the start of imaging.

  Furthermore, when the absolute time of the measurement start time, the measurement end time, the imaging start time, and the imaging end time is obtained as described later, an overlapping portion of the measurement end time from the measurement start time and the imaging end time from the imaging start time is obtained. Take out and associate the measurement data and image data of the overlapping part by the same ratio calculation.

  S15, S16: The server 10 transmits the measurement data and the image data to the display device 30. Specifically, a leader or the like requests measurement data and image data from the server 10 via the display device 30 or the like. Alternatively, a leader or the like may perform an operation of requesting measurement data and image data from the server 10 and the operation input receiving unit 14 of the server 10 may receive this operation. Alternatively, when the display device 30 inquires about the presence or absence of measurement data and image data by polling or the like and the measurement data and image data are transmitted to the server 10, the server 10 transmits the measurement data and image data to the display device 30 in response to the inquiry. May be.

  S17: The communication unit 31 of the display device 30 receives the measurement data and the image data, and the display control unit 32 displays the measurement data and the image data on the display 108 on the same time axis. Processing for display will be described with reference to FIG.

<< Operation Variation >>
FIG. 9 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 9, differences from FIG. 8 will be described. In FIG. 9, the trigger for starting measurement is different from that in FIG.

  S1: When starting measurement of walking motion, a leader or the like inputs an operation for starting measurement of walking motion to the display device 30. The operation input receiving unit 33 of the display device 30 receives an operation of a leader or the like.

  S1-1: The communication unit 31 of the display device 30 transmits a measurement start request to the server 10. The display device 30 and the server 10 can communicate by HTTP or the like. The measurement start request requests at least one of acceleration measurement start and pedestrian imaging start.

  S8: When the leader or the like determines that the pedestrian has sufficiently walked, the leader or the like inputs an operation to end the measurement of the walking motion to the display device 30. The operation input receiving unit 33 of the display device 30 receives an operation of a leader or the like.

  S8-1: The communication unit 31 of the display device 30 transmits a measurement end request to the server 10. The subsequent processing is the same as in FIG. The measurement end request is a request for at least one of the end of measurement of acceleration and the end of imaging of a pedestrian.

  Therefore, the instructor can request the server 10 to start and end the measurement from the display device 30 without operating the server 10. When a leader or the like is operating the display device 30, it is not necessary to move to the server 10.

<< Operation Variation >>
FIG. 10 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 10, the difference from FIG. 8 will be described. In FIG. 10, the start of measurement is that the pedestrian has started walking. The pedestrian starts walking and ends with the guidance of a leader or the like, for example. Alternatively, the circuit goes around the line from the start position on the line and stops at the start position.

  S1: The acceleration detector 52 of the acceleration measuring device 50 detects that a pedestrian has started walking based on the acceleration. For example, an acceleration exceeding a threshold value was detected in one or more of the x direction, the y direction, and the z direction, or a change in position per unit time obtained by integrating the acceleration twice exceeded the threshold value. The start of walking can be determined by, etc. The determination may be made based on the angular velocity detected by the gyro sensor 207, the position detected by the GPS receiver 214, and the direction detected by the acceleration / direction sensor 206. By detecting the start of walking, the acceleration detector 52 starts measuring acceleration (S6).

  S2: The communication unit 51 of the acceleration measuring device 50 transmits a walking start notification to the server 10. Thereby, the server 10 can transmit an imaging start request to the omnidirectional image capturing device 60. Note that the communication unit 51 of the acceleration measuring device 50 may transmit a walking start notification to the omnidirectional image capturing device 60 without going through the server 10.

  S8: The acceleration detector 52 of the acceleration measuring device 50 detects that the pedestrian has stopped walking based on the acceleration. For example, acceleration that is greater than or equal to the threshold is no longer detected in one or more of the x, y, and z directions, or the change in position in unit time obtained by integrating acceleration twice is less than the threshold, The end (stop) of walking can be determined by, for example. The determination may be made based on the angular velocity detected by the gyro sensor 207, the position detected by the GPS receiver 214, and the direction detected by the acceleration / direction sensor 206. By detecting the stop of walking, the acceleration detector 52 ends the measurement of acceleration.

  S9: The communication unit 51 of the acceleration measuring device 50 transmits a walk end notification to the server 10. Thereby, the server 10 can transmit an imaging end request to the omnidirectional image capturing device 60. Note that the communication unit 51 of the acceleration measuring device 50 may transmit a walking end notification to the omnidirectional image capturing device 60 without using the server 10. The subsequent processing may be the same as in FIG.

  According to the processing of FIG. 10, the measurement of walking motion can be started and ended without an instructor or the like performing an operation such as measurement start or measurement end.

<< Operation Variation >>
FIG. 11 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 11, differences from FIG. 8 will be described. In this embodiment, since it is effective to synchronize the time axes of measurement data and image data, in FIG. 8, the measurement start and imaging start, and the measurement end and imaging end are synchronized, respectively. Therefore, although absolute time is unnecessary, the server 10 may synchronize the measurement data and the image data using the absolute time as shown in FIG.

  S3-1: When the communication start part 51 of the acceleration measuring device 50 receives the measurement start request in step S2, the acceleration detecting part 52 of the acceleration measuring device 50 requests the server 10 for the start time.

  S3-2: Thereby, the acceleration detector 52 of the acceleration measuring device 50 can acquire the absolute time (measurement start time) at which the measurement is started from the server 10.

S5-1: When the communication unit 61 of the omnidirectional image imaging device 60 receives the imaging start request in step S4, the imaging unit 62 of the omnidirectional image imaging device 60 requests the server 10 for the start time.
S5-2: Thereby, the imaging unit 62 of the omnidirectional image imaging device 60 can acquire the absolute time (imaging start time) when imaging was started from the server 10.

S9-1: When the communication unit 51 of the acceleration measuring device 50 receives the measurement end request in step S9, the acceleration detecting unit 52 of the acceleration measuring device 50 requests the server 10 for the end time.
S9-2: Thereby, the acceleration detection unit 52 of the acceleration measuring device 50 can acquire the absolute time (measurement end time) when the measurement is ended from the server 10.

  S10-1: When the communication unit 61 of the omnidirectional image imaging device 60 receives the imaging end request in step S10, the imaging unit 62 of the omnidirectional image imaging device 60 requests the server 10 for the end time.

  S10-2: As a result, the imaging unit 62 of the omnidirectional image capturing apparatus 60 can acquire the absolute time (imaging end time) when the imaging is completed from the server 10.

  The server 10 acquires acceleration measurement start time and measurement end time, and pedestrian imaging start time and imaging end time together with measurement data and image data. Even if a time lag occurs between the start of acceleration measurement and the start of imaging for some reason, or a time lag occurs between the end of measurement of acceleration and the end of imaging, the server 10 can synchronize the time axes of the measurement data and the image data. it can.

<< Operation Variation >>
FIG. 12 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 12, differences from FIG. 11 will be described. In FIG. 11, the server 10 transmits a measurement start request, an imaging start request, a measurement end request, and an imaging end request to the acceleration measuring device 50 and the omnidirectional image capturing device 60, respectively. In FIG. 12, the server 10 communicates with the acceleration measuring device 50 and transmits a measurement start request and a measurement end request.

  S2-1: Upon receiving the measurement start request, the communication unit 51 of the acceleration measuring device 50 transmits an imaging start request to the omnidirectional image capturing device 60.

  S2-2: The communication unit 61 of the omnidirectional image imaging device 60 receives the imaging start request, and the imaging unit 62 of the omnidirectional image imaging device 60 requests the acceleration measuring device 50 for the start time.

  S2-3: When the communication unit 51 of the acceleration measuring apparatus 50 is requested for the start time, the communication unit 51 requests the server 10 for the start time.

  S2-4: In response to this, the server 10 transmits the measurement start time to the acceleration measuring device 50.

  S2-5: The acceleration measuring device 50 transmits the imaging start time to the omnidirectional imaging device 60. Therefore, the acceleration measuring device 50 and the omnidirectional image capturing device 60 can treat the same start time as the measurement start time and the image capturing start time, respectively.

  Note that the time (current time) held by the acceleration measuring device 50 at the time of step S2-2 may be transmitted to the omnidirectional imaging device 60. The acceleration measuring device 50 starts measuring the acceleration from this time, and the omnidirectional imaging device 60 starts imaging from this time, so that the measurement start time and the imaging start time can be synchronized.

  The processing at the end of measurement and imaging is the same.

  S9-1: Upon receiving the measurement end request, the communication unit 51 of the acceleration measuring device 50 transmits an imaging end request to the omnidirectional image capturing device 60.

  S9-2: The communication unit 61 of the omnidirectional image imaging device 60 receives the imaging end request, and the imaging unit 62 of the omnidirectional image imaging device 60 requests the acceleration measuring device 50 for the end time.

S9-3: When the communication unit 51 of the acceleration measuring apparatus 50 is requested for the end time, the communication unit 51 requests the server 10 for the end time.
S9-4: In response to this, the server 10 transmits the measurement end time to the acceleration measuring device 50.
S9-5: The acceleration measuring device 50 transmits the imaging end time to the omnidirectional image capturing device 60. Therefore, the acceleration measuring device 50 and the omnidirectional imaging device 60 can treat the same end time as the measurement end time and the imaging end time, respectively.

  Note that the time (current time) held by the acceleration measuring device 50 at the time of step S9-2 may be transmitted to the omnidirectional imaging device 60. The acceleration measuring device 50 finishes measuring the acceleration at this time, and the omnidirectional imaging device 60 finishes the imaging at this time, so that the measurement end time and the imaging end time can be synchronized.

<Display example of image data and measurement data>
A display example of image data and measurement data will be described with reference to FIG. FIG. 13 shows an example of image data and measurement data displayed on the display 108 by the display device 30. A screen as shown in FIG. 13 is referred to as a data display screen 501. The data display screen 501 mainly has an image data field 502 and a measurement data field 503. The display control unit 32 displays the image data in the image data column 502 and displays the measurement data in the measurement data column 503. The display control unit 32 extracts the pedestrian imaging range from the image data of the omnidirectional image and displays it in the image data field 502. This process will be described later.

  The image data column 502 will be described. In FIG. 13, accelerations in the x direction, y direction, and z direction are displayed in time series. An instructor or the like can display the acceleration in any one or more directions among the x direction, the y direction, and the z direction. When the image data is a still image, when an instructor or the like presses the button 504, the operation input receiving unit 33 receives the operation, and the display control unit 32 displays the next image data in the image data column 502 in terms of time. Similarly, the display control unit 32 can display the previous image data in the image data column 502 by operating the button. In addition, the display control unit 32 may display image data as a still image in the image data column 502 one after another at a constant time interval.

  When the image data is a moving image, when an instructor or the like presses the button 504, the operation input receiving unit 33 receives the operation, and the display control unit 32 reproduces the image data as a moving image at the same reproduction speed as the imaging speed (number of frames / sec). To do. High-speed playback and low-speed playback can be stopped. When a leader or the like presses the button 504 again, the operation input receiving unit 33 receives the operation, and the display control unit 32 stops the reproduction. Therefore, a leader or the like can display image data at an arbitrary timing. Note that not only the image data but also sound collected together with the image data may be output.

  The measurement data column 503 will be described. Whether the image data is a still image or a moving image, the measurement data is continuously measured. Alternatively, in the case of a still image, the acceleration measuring device 50 and the omnidirectional image capturing device 60 are synchronized so that the acceleration measuring device 50 acquires acceleration at the timing when the omnidirectional image of the still image is captured. Good.

  The display control unit 32 scales and displays the measurement data in the measurement data column 503 so that all of the measurement data can be displayed in the measurement data column 503. Alternatively, measurement data for a certain amount of time may be displayed in the measurement data column 503 and switched at regular time intervals. When measurement data having a certain length is displayed in the measurement data column 503 as shown in FIG. 13A, the display control unit 32 performs measurement associated with the image data displayed in the image data column 502. A bar 505 indicating data is displayed in the measurement data column 503. For example, when the measurement data for the whole or a certain time is displayed in the measurement data column 503 and the image data of the moving image is reproduced, the display control unit 32 is associated with the frame of the image data being displayed. A bar 505 is superimposed on the measurement data. Therefore, the bar 505 moves the measurement data column 503 to the right with time or as the reproduction of the moving image proceeds. Conversely, when an instructor or the like moves the bar 505, the operation input accepting unit 33 accepts the operation, and the display control unit 32 displays an image data frame associated with the measurement data indicated by the moved bar 505. It is displayed in the data column 502.

  Moreover, you may display measurement data like FIG.13 (b). In FIG. 13B, only the measurement data associated with the image data displayed in the image data column 502 and the measurement data before and after that are displayed in the measurement data column 503. In the measurement data column 503, for example, only measurement data of about 1 second to several seconds is displayed at the same time. Note that the dotted measurement data indicates that the measurement data is continuous and is not displayed on the screen.

  In the case of FIG. 13B, the display control unit 32 displays the measurement data associated with the omnidirectional image displayed in the image data column 502 in the center of the measurement data column 503 as the moving image or still image progresses. To display. Since the image data and the measurement data are associated with each other by the server 10, at least the measurement data associated with the frame of the image data displayed in the image data column 502 can be displayed in the measurement data column 503. . When the moving image or still image in the image data field 502 is stopped, the measurement data is also stopped.

<< Evaluation of measurement data >>
When the measurement data is acceleration, peaks appear periodically and the peak height does not fluctuate so much in general walking. On the other hand, if the walking function declines due to age or injury, the peak of acceleration does not appear periodically or the height of the peak tends to fluctuate. The instructor or the like may be able to evaluate the walking motion only by looking at the image data, but the measurement data can support that the walking motion is not appropriate. In addition, an instructor or the like can specify a place where the measurement data is disturbed and can check the image data. Although the measurement data has a list property, since the image data is displayed only at a frame at a certain moment, an instructor or the like can search the image data for an inappropriate walking motion in a short time.

  In this embodiment, the acceleration is described as an example. However, whether or not the walking motion is appropriate depending on the angular velocity by the gyro sensor 207, the position detected by the GPS receiver, the direction detected by the acceleration / direction sensor, and the like. Can be supported. For example, the angular velocity measured by the gyro sensor 207 indicates the speed of change of the yaw angle, pitch angle, and roll angle when a pedestrian walks. In this case as well, it is known that, as with acceleration, peaks appear periodically and the peak height does not fluctuate much in general walking. Further, since the acceleration is obtained by differentiating the position twice, the position also gives information equivalent to the acceleration. In the case of turning walking, the azimuth constantly changes, but acceleration of the change in azimuth can be obtained by differentiating the azimuth twice. In this case as well, the peak period and height are a guide for evaluation. In addition, in these measurement data, if the peak is periodic and the peak fluctuation is extremely large even if the height fluctuation is small, the instructor may indicate that there is a possibility of improvement in walking motion. Can be confirmed.

  Further, the image rotation unit 34 of the display device 30 may calculate the area between the acceleration and the acceleration = 0 axis. For example, if you are walking in a circular shape, the y direction is the acceleration in the left-right direction of the pedestrian (the radial direction of the circle), so the area on the positive side and the area on the negative side are the acceleration in the radial direction relative to the line. Show shake. If a pedestrian walks on the line, the area on the positive side will be approximately equal to the area on the negative side. Therefore, it can be understood that the balance between the left leg and the right leg is poor when the area on the positive side and the area on the negative side are greatly different.

  Thus, a leader or the like can obtain a lot of information by viewing the measurement data and the image data whose time axes are synchronized.

<Trimming pedestrians from spherical images>
Since the image data includes an image of 360 ° around, if the entire image is displayed, the image is distorted and the walking motion is difficult to understand. For this reason, it is preferable that the display control unit 32 displays only the range where the pedestrian is shown in the image data field 502. Such processing is expressed as trimming of the omnidirectional image, extraction of pedestrians, rotation of the omnidirectional image, or extraction. The rotation of the omnidirectional image refers to determining the range displayed in the image data field 502 by rotating the omnidirectional image horizontally (longitudinal direction).

  FIG. 14 is an example of a flowchart illustrating a procedure in which the moving object detection unit 35 specifies a range in which a pedestrian is shown and displays it in the image data field 502. The processing in FIG. 14 starts when, for example, the display device 30 receives image data and a moving image or a still image is played back.

  First, the moving body detection unit 35 specifies the position of the pedestrian's foot (S10). Foot refers to the part of the footwear, from the heel to the toe when not wearing footwear. However, it may be detected as a foot below the waist or below the knee. In the case of a moving image, if the pedestrian performs an action that steps several times, the moving object detection unit 35 can detect the pedestrian by detecting the moving object. Or a pedestrian can be detected even if a face is recognized. There is a possibility that a person other than the pedestrian exists around the omnidirectional image capturing device 60, but the moving body occupying the largest pixel range is detected as a pedestrian. The pixel with the lowest latitude among the moving objects is the foot position. In addition, the moving object detection unit 35 may learn a foot image by machine learning and recognize the foot from the omnidirectional image.

  Further, assuming that the pedestrian starts walking from a predetermined position, the initial position of the pedestrian is a predetermined position in the horizontal direction of the omnidirectional image. Therefore, in this case, the moving object detection unit 35 can easily identify the pedestrian to be measured.

  Once the pedestrian can be identified from the omnidirectional image, the position of the pedestrian in the subsequent omnidirectional image should be able to be detected from the vicinity of the identified pedestrian, and can be easily detected in a relatively short time.

  In the case of a still image, the position of the pedestrian in the initial omnidirectional image is determined or input by a leader or the like. Or a pedestrian can be detected even if a face is recognized. A pedestrian can be detected from each still image by performing pattern matching on the subsequent omnidirectional image using the circumscribed rectangle of the pedestrian in the initial omnidirectional image.

  When the pedestrian starts walking, the moving body detection unit 35 calculates the angular velocity of the pedestrian per unit time from the change in the position of the foot (S20). The angular velocity will be described with reference to FIG. FIG. 15 is a diagram schematically showing the position of the pedestrian's foot 511 in the top view of the omnidirectional image. For example, assume that the north direction is the reference. The foot 511 can take a position θ in the range of 360 ° in the longitude direction (an angle with the north facing zero degree) θ. Since two legs 511 are detected in one frame, the average is set as the position of the legs 511. For example, the average change amount of the foot 511 in the past several seconds is calculated every predetermined time (for example, every time the frame is updated). Thereby, the angular velocity of a pedestrian with little fluctuation | variation can be calculated. The angular velocity can be referred to as a walking speed (moving speed) viewed from the omnidirectional image capturing device 60. The walking speed in the x direction in real space is obtained by integrating the acceleration in the x direction.

  The image rotation unit 34 rotates the image at the angular velocity calculated in step S20 (S30). That is, when the frame is switched for the reproduction of the moving image, the image rotation unit 34 trims the image in the display range (predetermined range) of the image data column 502 around the position where the pedestrian moves at the angular velocity. Thereby, the pedestrian is reflected in the center of the image taken out by the image rotation unit 34.

  The display control unit 32 displays the image in the display range trimmed by the image rotation unit 34 in the image data field 502. Therefore, a leader or the like can evaluate the walking motion of the pedestrian without searching for the pedestrian from the omnidirectional image.

  In addition, since a camera with a normal angle of view is captured so that a pedestrian can be seen, processing such as rotation and trimming is not necessary. The server 10 or the omnidirectional image capturing device 60 may have the functions of the moving object detection unit 35 and the image rotation unit 34. In this case, communication load and communication time when image data is transmitted from the server 10 or the omnidirectional image capturing device 60 to the display device 30 can be reduced.

<< Rotating spherical images by face recognition >>
In addition to paying attention to the position of the foot, the image rotation unit 34 may recognize a pedestrian by recognizing a face from the omnidirectional image and rotate the omnidirectional image.

  FIG. 16 is an example of a flowchart illustrating a procedure in which the moving object detection unit 35 specifies a range in which a pedestrian is shown and displays it in the image data field 502. The processing of FIG. 16 starts when, for example, the display device 30 receives image data and a moving image or a still image is reproduced.

  First, the moving object detection unit 35 recognizes a face for each omnidirectional image (S10). FIG. 17 is an example of a diagram for schematically explaining the recognition of the face 512. The face 512 is recognized by setting an appropriate feature amount and learning by a learning identification device. For example, as feature quantities, Haar-like features, LBP (Local Binary Patterns) features, HOG (Histogram of Oriented Gradients) features, and the like are known. As learning methods of the learning identification device, SVM (Support Vector Machines), AdaBoost, and the like are known. However, the present invention is not limited to these, and it is sufficient that the face 512 can be recognized.

  The image rotation unit 34 trims the image in the display range 513 (predetermined range) in the image data field 502 around the face 512 from the omnidirectional image. Thereby, the pedestrian is reflected in the center of the image taken out by the image rotation unit 34.

  The display control unit 32 displays the image in the display range 513 trimmed by the image rotation unit 34 in the image data field 502. Therefore, a leader or the like can evaluate the walking motion of the pedestrian without searching for the pedestrian from the omnidirectional image.

  Instead of determining and trimming the position of the face 512 for each omnidirectional image, the image rotation unit 34 calculates the moving speed in the longitude direction from the positions of the faces 512 of several past omnidirectional images. Thus, the display range may be determined from the omnidirectional image to be displayed next. Since the position of the face 512 is less likely to fluctuate, a smooth moving image can be displayed.

<Detection of moving objects using multiple spherical images>
In continuous spherical omnidirectional images captured at time intervals that can be regarded as moving images or moving images, a moving object can be detected using a plurality of temporally continuous omnidirectional images.

  First, an outline of detection of a moving object will be described with reference to FIG. FIG. 18 is an example of a diagram that schematically illustrates detection of a moving object.

  In FIG. 18A, a frame image 181 of interest and a frame image 182 immediately before it are displayed. If the frame rate is 30 fps, for example, there is an interval of approximately 33 [milliseconds] between the frame image 181 of interest and the immediately preceding frame image 182. Instead of the target frame image 181 and the immediately preceding frame image 182, a frame image several frames before the target frame image 181 may be adopted. The moving object detection unit 35 calculates the difference between the frame image of interest and the immediately preceding frame image. Difference means subtracting pixel values of pixels at the same position. Even if the exact same subject is shown (even if there is no moving object), the difference may not become zero due to blinking of fluorescent light, movement of a distant person, change of shadow, etc., so if the difference is less than the threshold, it will be zero Consider.

  FIG. 18B shows an example of the difference image. In FIG. 18B, pixels whose difference is greater than or equal to the threshold are replaced with white pixels. Therefore, when comparing the frame image of interest with the immediately preceding frame image, there is a high possibility that a pedestrian exists so as to overlap the white pixel.

  Next, as illustrated in FIG. 18C, the moving object detection unit 35 performs pixel expansion processing on white pixels a plurality of times. Pixel expansion processing refers to replacing 8 pixels (or 4 pixels adjacent vertically and horizontally) in contact with white pixels with white pixels. Since the pixel in contact with the white pixel is traced by a plurality of processes, the white pixel appears to expand as the number of times increases. Multiple times is a fixed number such as 8 times or 10 times, and a suitable value is experimentally determined. Alternatively, the pixel expansion process may be stopped when the area of the white pixel exceeds a certain level. Immediately after the difference is calculated, the scattered white pixels become a block having a size. In the case of a pedestrian, since the difference image is vertically long, white pixels are enlarged in the vertical direction by the pixel expansion process, and a slightly larger range (a slightly larger range than the pedestrian) can be specified.

  Next, as shown in FIG. 18D, a circumscribed rectangle 601 or the like of a region where white pixels are continuous is specified. The image rotation unit 34 trims the display range from the omnidirectional image so that the coordinates (center, center of gravity, etc.) of the circumscribed rectangle 601 detected by the above processing are in the center. Since there is a possibility that moving objects other than pedestrians exist around the omnidirectional image capturing device 60, when a plurality of circumscribed rectangles 601 are detected, the circumscribed rectangle 601 having the largest area of the circumscribed rectangle 601 is displayed. adopt.

<< Trimming from spherical image >>
FIG. 19A is an example for explaining a display range that is trimmed from the omnidirectional image. The omnidirectional image is a sphere, but can be transformed into a rectangle as shown in FIG. 19 by Mercator projection or the like. Therefore, it has a width of 360 ° in the longitude direction (horizontal direction, X direction) and 180 ° in the latitude direction (vertical direction, Y direction). It is assumed that the longitude at which the coordinate in the X direction is zero is determined.

When the circumscribed rectangle 601 is obtained as shown in FIG. 19A, the coordinates of the four corners of the circumscribed rectangle 601 are also apparent, so the image rotation unit 34 determines the center O (x ₀ , y ₀ ) of the circumscribed rectangle 601. Can be determined. The image rotation unit 34 trims a predetermined range in the longitude direction around the center O, and trims a predetermined range in the latitude direction around the center O. This range is the display range displayed in the image data column 502. In FIG. 19, the range of the dotted line is the display range 513 to be trimmed. Since the height of the pedestrian does not vary much, the display range in the latitude direction may be a fixed value. In this case, there is an advantage that the pedestrian is less likely to fluctuate when displayed on the display device 30.

  When the pedestrian walks along the line L that is away from the omnidirectional image capturing device 60, the circumscribed rectangle 601 becomes smaller. Conversely, when approaching the omnidirectional image capturing apparatus 60, the circumscribed rectangle 601 becomes larger. Therefore, it is preferable to adjust the display magnification (zoom in / zoom out) according to the size of the circumscribed rectangle 601. For example, the size of a fixed circumscribed rectangle 601 is determined in advance, and the display magnification is determined according to the ratio with the detected size of the circumscribed rectangle 601. For example, the ratio of the area, height, and width of the circumscribed rectangle 601 is calculated. By scaling the omnidirectional image using this ratio as a display magnification, the size of the displayed pedestrian can be made almost constant and it is easy to view.

  By the way, when the pedestrian moves, it reaches the end in the longitude direction (0 ° or 360 °), enters from the other end (360 ° or 0 °), and is imaged again. In this case, a transient state in which one moving body is detected at both of the two end portions may occur.

  FIG. 19B is an example of a diagram illustrating a case where a moving object is detected at both of two ends. It is assumed that the pedestrian is moving in the direction of decreasing longitude. When a pedestrian approaches the end, a moving object is detected at two locations, an end on the 0 ° side and an end on the 360 ° side. In order to determine where the pedestrian is, it may be considered to give priority to the larger size of the circumscribed rectangle 601. For example, in FIG. 19B, when the area of the circumscribed rectangle 601a is larger than the area of the circumscribed rectangle 601b, the moving object detection unit 35 determines the center Oa of the circumscribed rectangle 601b as the center of the pedestrian.

  When the pedestrian moves further, the area of the circumscribed rectangle 601b becomes larger than the area of the circumscribed rectangle 601a. The moving body detection unit 35 determines that the center Ob of the circumscribed rectangle 601b is the center of the pedestrian. Accordingly, the center O of the circumscribed rectangle 601 can be determined even when the pedestrian walks across the end, but the display range 513 is instantly determined when the area of the circumscribed rectangle 601b is larger than the area of the circumscribed rectangle 601a. Will move greatly. This is because the center O of the circumscribed rectangle 601 is always inside the end. In addition, when the size relationship between the two circumscribed rectangles 601a and 601b at the end portions does not change uniformly, the position of the display range 513 is not stable (in one frame, the area of the circumscribed rectangle 601b is larger than the area of the circumscribed rectangle 601a). In the next frame, it is determined that the area of the circumscribed rectangle 601a is larger than the area of the circumscribed rectangle 601b). If the position of the display range 513 is not stable, the position of the display range 513 may be shifted in the longitude direction for each frame.

Therefore, in the present embodiment, the center O of the circumscribed rectangle 601 is determined by the following two methods.
I. Method of always monitoring two omnidirectional images FIG. 20 is an example of a diagram illustrating a method of determining the center O of the circumscribed rectangle 601 using two omnidirectional images. The omnidirectional image A (first image data) shows up to 360 ° with 0 ° as an end. This 0 ° is determined as a predetermined reference position of the omnidirectional image captured by the omnidirectional image capturing device 60. On the other hand, the omnidirectional image B (second image data) shows 180 ° up to 180 °. That is, in the omnidirectional images A and B, the cut start position is shifted by 180 ° in the longitude direction. The cut end position is the same as or just before the start position.

  The moving object detection unit 35 conventionally performs image processing using the omnidirectional image A, but creates the omnidirectional image B in this embodiment. The moving object detection unit 35 can detect the circumscribed rectangles 601a and 601b from the omnidirectional image A and the circumscribed rectangle 601c from the omnidirectional image B, respectively. Then, the circumscribed rectangle 601c having the largest area is specified, and the center Oc of the circumscribed rectangle 601c is used for the display range 513. If the center O in the longitude direction is determined, the display range 513 may be trimmed from either the omnidirectional image A or B. Thereby, even if a pedestrian approaches the end, blurring of the display range 513 is suppressed and a stable display range 513 is obtained.

  When the pedestrian is not approaching the end, the omnidirectional images A and B are the same, so the area of the circumscribed rectangle 601 is the same. When the area of the circumscribed rectangle 601 is the same, the center O of the circumscribed rectangle 601 may be determined by either the omnidirectional image A or B.

  FIG. 21 is an example of a flowchart of a procedure in which the display device 30 always monitors two omnidirectional images and trims the display range 513. The process of FIG. 21 is repeatedly executed for each frame.

  The display control unit 32 creates an omnidirectional image A by converting the omnidirectional image by Mercator projection or the like. The moving object detector 35 reduces the omnidirectional image A in order to detect moving objects using the omnidirectional image A (S10). Although reduction is not an indispensable process, the size can be reduced by reduction, so that the memory consumption can be suppressed and the time for image processing can be shortened. In particular, since the omnidirectional image is large in size, it is effective to reduce it. An image with a certain number of pixels is sufficient for detecting a moving object.

  The moving object detection unit 35 generates an omnidirectional image B that is 180 ° different from the omnidirectional image A in the cutting start position, and similarly reduces (S20).

  The moving object detection unit 35 detects moving objects in the omnidirectional images A and B, and calculates the area of the circumscribed rectangle 601 (S30). When a plurality of circumscribed rectangles 601 are detected in each of the omnidirectional images A and B, the largest circumscribed rectangle 601 is determined from each of the omnidirectional images A and B.

  The moving object detection unit 35 determines whether or not the area of the circumscribed rectangle 601 is the same in the omnidirectional images A and B (S40).

  If the determination in step S40 is Yes, since the pedestrian is not approaching the end, the moving object detection unit 35 determines the center O of the circumscribed rectangle 601a of the omnidirectional image A (S50). Note that the center O of the circumscribed rectangle 601b of the omnidirectional image B may be determined.

  If the determination in step S40 is No, there is a possibility that the pedestrian is approaching the end, so the moving object detection unit 35 determines the center of the circumscribed rectangle 601 having the largest area (S60). As a result, the circumscribed rectangle 601 that is not the end of the omnidirectional images A and B is selected, so that the center O of the circumscribed rectangle 601 can be stably calculated.

  Subsequent to step S50 or S60, the image rotation unit 34 trims the display range 513 based on the center O of the circumscribed rectangle 601 (S70).

  Through the processing as described above, the display device 30 can display the display range 513 with less blurring in which the pedestrian is reflected in the center.

II. Method for switching omnidirectional image when pedestrian approaches end part FIG. 22 is an example of a diagram for explaining a method for switching an omnidirectional image when a pedestrian approaches the end part. The omnidirectional images A and B are the same as those in FIG.

  First, it is assumed that a pedestrian is walking near the center of the omnidirectional image A as shown in FIG. The moving object detection unit 35 can detect a pedestrian from the omnidirectional image A. Next, as shown in FIG. 22B, when the pedestrian approaches the end of the omnidirectional image A, the moving object detection unit 35 detects the omnidirectional image A from the omnidirectional image A as shown in FIG. A celestial sphere image B is generated. Whether or not the vehicle has approached can be determined based on whether or not the distance from the end to the center O of the circumscribed rectangle 601 is less than the threshold value K. Alternatively, the distance between the left end or the right end of the circumscribed rectangle 601 and the end may be compared with the threshold value K.

  As shown in FIG. 22C, since the omnidirectional image B has a start position of clipping different by 180 °, the pedestrian of the omnidirectional image B appears near the center. Therefore, when the moving object detection unit 35 detects a pedestrian from the omnidirectional image B, the center O of the circumscribed rectangle 601 with less blur can be determined. Thereafter, when the pedestrian walks and approaches the end of 180 ° and the distance becomes less than the threshold value K, the moving object detection unit 35 switches to the omnidirectional image A.

  The start position of the cut out of the omnidirectional image B is an example, and may be cut out at 90 ° to 90 °, for example. In this case, since the space in the advancing direction is longer than that of cutting at 180 ° to 180 °, switching frequency of the cutting start position can be reduced. In addition, the cut start position may be determined according to the threshold value K. For example, the cut start position is determined by shifting the longitude slightly larger than the threshold value K. In this case, the cut start position can be optimally determined with respect to the threshold value K.

  FIG. 23 is an example of a flowchart of a procedure in which the omnidirectional image is switched and the display device 30 trims the display range 513 when a pedestrian approaches the end. The process of FIG. 23 is repeatedly executed for each frame.

  First, the moving object detection unit 35 determines whether or not the center O of the previous circumscribed rectangle 601 is less than the threshold value (less than the threshold value K) from the end (S10). The position of the center O may be acquired from either the omnidirectional image A or B. The previous time refers to the center O of the circumscribed rectangle 601 detected in the frame image one frame before. When the pedestrian travel direction is fixed, the distance and the threshold need only be compared with either the 0 ° or 360 ° end. When the traveling direction of the pedestrian is not determined in a certain direction, the traveling direction of the pedestrian is considered as the distance from the end to the center O. The traveling direction of the pedestrian can be determined by the sign of the difference between the centers O (longitude directions) of the past two frames or more. For example, in the case of the traveling direction approaching the 0 ° end, the distance between the center O and the threshold may be compared only with the 0 ° end.

  When the determination in step S10 is Yes, the moving object detection unit 35 changes the start position for cutting the omnidirectional image (S20). Therefore, if the omnidirectional image A was used last time, it is switched to the omnidirectional image B, and if the omnidirectional image B was used last time, it is switched to the omnidirectional image A. When the display device 30 automatically generates the omnidirectional image A and the omnidirectional image A is in the cutout range, it is not necessary to generate the omnidirectional image A. When the display device 30 does not automatically generate the omnidirectional image A and the omnidirectional image A is in the cutout range, the omnidirectional image A is generated. When the omnidirectional image B is in the cut-out range, the omnidirectional image B is generated.

  If the determination in step S10 is No, the process proceeds to step S30, so that the clipping range of the omnidirectional image is not changed.

  Next, the moving object detection unit 35 reduces the cut-out omnidirectional image (S30). The effect of reduction has been described with reference to FIG.

  Next, the moving object detection unit 35 determines the center O of the circumscribed rectangle 601 of the omnidirectional image A or B (S40). Then, the image rotation unit 34 trims the display range 513 (S50).

  According to such processing, blurring of the display range 513 can be suppressed, and the processing load can be reduced because the center O of the circumscribed rectangle 601 can be determined from either the omnidirectional image A or B.

<When there are multiple pedestrians>
A case where a plurality of pedestrians are walking around the omnidirectional image capturing device 60 will be described. When a pedestrian to be monitored by a leader or the like is determined among a plurality of pedestrians, the display device 30 may detect only a pedestrian designated by the leader or the like. For example, the display device 30 receives the position of the pedestrian designated by the instructor, and determines that the target pedestrian is reflected in the circumscribed rectangle 601 having the closest position from the omnidirectional images having different imaging times.

  However, when a plurality of pedestrians are imaged, the circumscribed rectangles 601 of the two pedestrians may be superimposed due to differences in walking speeds and walking directions. For this reason, it is effective to estimate the walking speed of the pedestrian as follows.

  FIG. 24 is an example of a diagram illustrating the center O of the circumscribed rectangles 601A and 601B when two pedestrians are imaged. 24A shows the omnidirectional image (frame image) at time t-2, FIG. 24B shows the omnidirectional image at time t-1, and FIG. 24C shows the omnidirectional image at time t-0. Each celestial sphere image is shown. Two pedestrians are captured in each spherical image. In this case, the moving object detection unit 35 determines not only the circumscribed rectangle 601 having the largest area but also the circumscribed rectangle 601 whose area is equal to or larger than the threshold value. Thereby, a plurality of pedestrians can be detected.

  Although the omnidirectional images from time t-0 to t-2 are captured at different times, the pedestrian of the omnidirectional image at the time when the time difference is slight is the position in the omnidirectional image at the next time. Can be determined to be the nearest pedestrian. Therefore, the moving object detection unit 35 can label the pedestrians A and B on the circumscribed rectangle 601 whose area is equal to or larger than the threshold. Further, the moving object detection unit 35 can calculate the walking speed (including direction) of the pedestrians A and B from the difference between the centers O of the circumscribed rectangles 601 of at least two past frame images. Accordingly, the positions of the pedestrians A and B in the longitude direction can be predicted in the omnidirectional image at time t + 1. Thereby, even when the pedestrians A and B overlap, it becomes easy to accurately determine the center O of each circumscribed rectangle 601A and 601B.

FIG. 25 is an example of a diagram illustrating the center O of the circumscribed rectangle when pedestrians overlap. In FIG. 25A, circumscribed rectangles 601A and 601B of pedestrians A and B are detected separately. In addition, walking speeds VA and VB of pedestrians _A and _B are calculated from past omnidirectional images.

In FIG. 25 (b), the circumscribed rectangles of pedestrians A and B overlap, and one circumscribed rectangle 601 is detected. In this case, the moving object detection unit 35, a pedestrian A, walking speed _V A of B, and using _{V B,} Pedestrian A, B respectively of the circumscribed rectangles 601A, the center of 601B _O A, calculates the _{O B.} For example, the center O _A of the circumscribed rectangle 601A is O _A + V _A · Δt, and the center O _B of the circumscribed rectangle 601B is O _B + V _B · Δt. Δt is the imaging interval of the omnidirectional image. By doing so, even if the circumscribed rectangles 601A and 601B of the pedestrians A and B overlap, the position (center O) of each pedestrian can be detected with relatively high accuracy.

  FIG. 26 is an example of a flowchart of a procedure in which the display device 30 identifies the positions of a plurality of pedestrians and trims the display range 513. The process of FIG. 26 is repeatedly executed for each frame.

  First, the moving object detection unit 35 calculates the circumscribed rectangle 601 and the center O (S10). At this time, the number of circumscribed rectangles 601 is not examined. When a plurality of circumscribed rectangles 601 are detected, the respective centers O are calculated.

  Next, the moving object detection unit 35 determines whether or not a plurality of pedestrians have been detected in the past frame (S20).

  If the determination in step S20 is no, the process proceeds to step S60. That is, since there is always one pedestrian, the display range 513 is trimmed as described above.

  When the determination in step S20 is Yes, the moving object detection unit 35 determines whether or not the number of pedestrians has decreased (S30). When the number of pedestrians decreases, the circumscribed rectangle 601 may overlap.

  For this reason, when determination of step S30 is Yes, the moving body detection part 35 calculates the center O with each walking speed of several pedestrians (S40). That is, the center O is estimated not by the center O in step S10 but by the walking speed.

  When the determination in step S30 is No, since the circumscribed rectangles 601 of a plurality of pedestrians are specified separately, the moving object detection unit 35 calculates each walking speed (S50).

  Following step S20, S40, or S50, the image rotation unit 34 trims the display range 513 (S60). Therefore, when the process passes through step S40, the display range 513 can be trimmed based on the center O estimated in step S40. When passing through step S50, the display range 513 can be trimmed based on the center O calculated in step S10.

  By such processing, even if a plurality of pedestrians are walking around the omnidirectional image capturing device 60, the display range 513 of each pedestrian can be trimmed.

  Although the turning direction is not described in detail in the first embodiment, when the instructor or the like analyzes how to walk, not only turning left or right but also turning right (first turning direction) and turning left It may be effective to compare images of (second turning direction). This is because when the function of the leg portion (one or both) is deteriorated, the way of walking may be affected depending on which leg is inside the turning direction. However, when the pedestrian turns to the right and to the left, the moving direction of the pedestrian in the image is reversed, and the direction of the body is also reversed, which makes it difficult to compare.

  In the present embodiment, an image display system 100 that allows an instructor or the like to easily compare a right turn image and a left turn image will be described.

  FIG. 27 is an example of a functional block diagram of the display device 30. In FIG. 27, the server 10, the omnidirectional image capturing device 60, and the acceleration measuring device 50 are omitted. Further, in the present embodiment, the components denoted by the same reference numerals in FIG. 6 perform the same function, and therefore, only the main components of the present embodiment may be mainly described.

  The display device 30 of this embodiment has a left / right reversing unit 36. The left / right reversing unit 36 is realized by a command from the CPU 101 shown in FIG. 3, and the left / right reversing unit 36 performs a left / right reversing process on the display range 513 trimmed by the image rotating unit 34.

<Right turn, left turn>
FIG. 28 is an example for explaining a pedestrian turning. FIG. 28A shows a pedestrian turning right, and FIG. 28B shows a pedestrian turning left. In FIG. 28A, since a pedestrian moving in the right direction is imaged in the image data, the right shoulder appears in front. In FIG. 28B, since a pedestrian moving in the left direction is imaged in the image data, the left shoulder appears in front. The procedure until the image data and the measurement data are transmitted to the display device 30 is described with reference to FIG. The right turn image data and measurement data and the left turn image data and measurement data may be transmitted simultaneously or separately. However, it is preferable that both are associated with information (for example, a pedestrian ID) indicating that they are the same pedestrian. Further, the file name and attribute information of the image data are set so that the right turn and the left turn are distinguished.

  In the right turn and the left turn, the pedestrian may start walking from the same position or may start walking from another position. The moving distance is approximately one turn of the same line L, but the imaging time of the image data may vary depending on the walking speed of the pedestrian. This is because many pedestrians have different walking speeds when turning right and left. Alternatively, when the server 10 or the like walks the pedestrian for the same amount of time during the right turn and the left turn, the imaging of the image data and the measurement data may be discontinued.

<Invert image>
FIG. 29 shows an example of a data display screen 501 displayed on the display 108 by the display device 30. Although the measurement data is not displayed in FIG. 29, the measurement data can also be displayed when the user performs a predetermined operation. FIG. 29A shows image data before left-right reversal, and FIG. 29B shows image data obtained by left-right reversing left-turn image data. In FIG. 29 (a), the direction of the pedestrian's body is reversed between the right turn and the left turn. On the other hand, in FIG. 29B, since the image data of the left turn is reversed left and right, the pedestrian's body direction is the same for the right turn and the left turn. Therefore, it is easy for leaders to compare. Note that the right-turn image data may be reversed horizontally. That is, only one of the right-turn image data and the left-turn image data needs to be horizontally reversed.

  The button 504 is the same as in FIG. The user can play back or stop the image data of the right turn or the left turn by pressing a button 504. Note that the button 504 may be set to one so that the image data of each of the right turn and the left turn can be reproduced and stopped by one operation. Alternatively, when either the right turn or left turn button 504 is pressed, the image data of the right turn and the left turn may be reproduced and stopped.

  Further, the user may be able to select whether or not to reverse the image data and the image data to be reversed.

  FIG. 30 is an example of a diagram illustrating inversion of image data. A known technique may be used to invert the image data. For example, the left / right reversing unit 36 exchanges the left and right pixel columns of the image data. That is, the pixel columns Ci (i is a pixel number from the center) having the same distance from the center are exchanged. The pixel column on the left side of the distance Ci is moved to the distance Ci on the right side, and the pixel column on the right side of the distance Ci is moved to the distance Ci on the left side. By repeating this from the center to the left end and the right end, the left-right inversion unit 36 can invert the image data left-right. Note that the illustrated horizontal reversal method is merely an example, and the horizontal reversal may be performed in any manner.

  FIG. 31 is an example of a flowchart illustrating a procedure in which the display device 30 displays the image data field 502 in a horizontally reversed range in which a pedestrian is shown. FIG. 31 mainly explains differences from FIG. The processing in steps S10 to S30 may be the same as in FIG. However, both the left turn and the right turn are performed.

  When the image is rotated in step S30, the left / right reversing unit 36 reverses the left / right of the left turn or right turn image data (S35). Specifically, only the display range 513 needs to be reversed left and right.

  Thereby, the display control part 32 can display the image data by which a person is always displayed in the center of an angle of view on the display 108 (S40).

<Indication of walking position>
In this embodiment, since the display range 513 is trimmed from the image data, it is difficult for a leader or the like to specify the position of the pedestrian. On the other hand, instructors may want to know how much the walking speed differs between right turn and left turn, and when there is an obstacle on the walking line L, the person wants to know the position of the pedestrian with respect to the obstacle. There are also. Therefore, it is effective to display pedestrian position information together with image data.

  FIG. 32 is an example of a diagram illustrating the position of a pedestrian. For the sake of explanation, it is assumed that the position of the pedestrian is specified by an angle θ with reference to a predetermined position of a circle centered on the omnidirectional image capturing device 60. This predetermined position is set as a reference position F. The pedestrian starts walking from this reference position F. Therefore, even if the elapsed time from the start of walking is the same, the position θ of the pedestrian varies depending on the walking speed. Note that since the position θ with respect to the reference position F only needs to be known, the pedestrian does not have to start walking from the reference position F. The position θ is measured clockwise (clockwise) from the reference position F.

  As described in the first embodiment, the image rotation unit 34 detects the position and face of the pedestrian's feet from the image data. Therefore, the center of gravity of the foot or face detected from the image data imaged 360 ° in the horizontal direction can be detected as the position θ of the pedestrian. Alternatively, the center of gravity of the circumscribed rectangle 601 may be used. Since the image rotation unit 34 notifies the display control unit 32 of the position θ detected from each frame of the image data, the display control unit 32 can display the position θ together with the image data.

  FIG. 33 is a diagram for explaining a data display screen 501 on which the position of a pedestrian is displayed. The data display screen 501 in FIG. 33 has a position display field 506 in each turning direction. In the position display field 506, a route shape icon 508 representing a line L and a position icon 507 are displayed. The display control unit 32 displays a position icon 507 at the position θ detected by the image rotation unit 34 on the route shape icon 508. As a result, the position icon 507 moves the path shape icon 508 as the image data is reproduced. A leader or the like can grasp where the user is walking on the line L. For example, the position icon 507 rotates clockwise when turning right, and the position icon 507 rotates counterclockwise when turning left. Therefore, even if the image data is inverted, the leader can determine the turning direction.

  On the other hand, when the image data is inverted, the position display field 506 may also be inverted. In this case, although the position of the pedestrian in the actual line L and the position of the position icon 507 on the route shape icon 508 are different, the direction in which the pedestrian moves in the image data field 502 and the moving direction of the position icon 507 ( (Direction of rotation) can be matched.

  FIG. 34 is an example of a diagram illustrating a method for determining the position of the position icon 507 to be reversed. It is assumed that the pedestrian is turning left. The actual position of the pedestrian (position θ detected by the image rotation unit 34) is θ clockwise from the reference position F. Therefore, θ ′ can be calculated from θ.

θ ′ = 360−θ
Since the counterclockwise position θ ′ from the reference position F coincides with the position θ, in order to invert the position icon 507, the position icon 507 may be moved clockwise from the reference position F by θ ′. The display control unit 32 displays a position icon 507 at this θ ′.

  FIG. 35 shows an example of a data display screen in which the position icon 507 is inverted. In FIG. 35, since the position icon 507 is substantially the same for the right turn and the left turn, the pedestrian can determine that the pedestrian has substantially the same walking speed for both the right turn and the left turn. In other words, it becomes easy for a leader or the like to find a pedestrian whose walking speed is different between right turn and left turn.

  FIG. 36 is an example of a flowchart for explaining a procedure for the display control unit 32 to display the data display screen 501. The processing in FIG. 36 is repeatedly executed for each frame of image data.

  First, the display control unit 32 displays a person in the center using the processing of FIGS. 14 and 16 (S10).

  Next, the image rotation unit 34 detects the position θ of the pedestrian in each of the right turn and the left turn (S20).

  Next, the display control unit 32 converts the pedestrian position θ into θ ′ in the horizontally reversed image data (S30).

  The display control unit 32 displays the position icon 507 in the position display field 506 of the image data for each of the right turn and the left turn (S40). That is, the position icon 507 is displayed at the position θ in the position display field 506 of the image data that has not been horizontally reversed. A position icon 507 is displayed at the position θ ′ in the position display field 506 of the image data reversed left and right.

<When a straight line is included in the route>
Since the line L is a curve, it is expected that a leader or the like can easily find a pedestrian who easily loses the center of gravity when turning. However, for some pedestrians, it may be effective to compare linear walking and turning walking. For example, if a pedestrian who walks without any trouble in a straight walk and has trouble with a walk in a turning walk, the leader or the like can determine that the center of gravity is easily broken, for example. On the other hand, in the case of a pedestrian who has difficulty in walking in both linear walking and turning walking, it can be determined that the leader or the like is likely to be due to physical characteristics.

  Therefore, it may be effective that the line L includes a straight portion and a turning portion. It is also effective for the leaders to compare the right turn and the left turn even in such a line L, but the straight parts of the right turn and the left turn are compared, and the right turn and the left turn It is preferable to compare the parts. However, if the walking speed of the pedestrian is different between the right turn and the left turn, the instructor or the like may compare the image data of the turning walk with the image data of the straight walking.

  Therefore, the display device 30 displays the image data of the right turn and the left turn in synchronization with each other between the straight part and the turning part.

  FIG. 37 is an example of a diagram illustrating a line L including a shape divided into a linear portion and a turning portion. In FIG. 37, points P1 to P2, P3 to P4 are straight portions, and points P2 to P3 and P4 to P1 are turning portions. In this way, the line L is divided by the points P1 to P4. For example, assuming that the point P1 is the reference position F, a pedestrian turning right walks in the order of points P1, P2, P3, and P4. Further, when turning left, assuming that the point P4 is the reference position F, a pedestrian who turns left walks in the order of points P4, P3, P2 and P1. The reason why the reference position F differs between the right turn and the left turn is to compare the straight portions and the turn portions in the right turn and the left turn.

  Since the points P1 to P4 are fixed, the respective positions θ1, θ2, θ3, and θ4 are known. For example, when the right turn pedestrian first reaches P2, the display control unit 32 stops the reproduction of the image data of the right turn pedestrian until the left turn pedestrian reaches P3. The same applies to the other points P3 to P1 of the right turn and the other points P2, P1 and P4 of the left turn.

  Conversely, when the left turn pedestrian first reaches P3, the display control unit 32 stops the reproduction of the image data of the left turn pedestrian until the right turn pedestrian reaches P2. The same applies to the other points P3 to P1 of the right turn and the other points P2, P1 and P4 of the left turn. By doing so, the display device 30 displays the image data of the right turn and the left turn in synchronization with each other between the straight portion and the turning portion.

  In FIG. 37, since the reference position F is different, the calculation method of the position θ ′ when the position icon 507 is reversed is different. That is, since the position θ4 when turning left is the position θ1 (0 °) when turning right, the position θ ′ of the inverted position icon 507 is calculated as follows.

θ ′ = 270−θ
FIG. 38 is an example of a diagram illustrating a data display screen 501 on which image data while walking on a line L including a straight line portion and a turning portion is displayed. In FIG. 38, the left turn image data is reversed left and right, and the left turn position icon 507 is also reversed and displayed.

  The upper image data in FIG. 38A is the image data of the straight line portion from the points P1 to P2, and the lower image data in FIG. 38A is the image data of the straight line portion from the points P4 to P3. Since both are linear walks, as can be seen from the position of the position icon 507, the walking speed of this pedestrian is the same whether turning right or turning left. Therefore, this pedestrian reaches point P2 (in the case of a right turn) and point P3 (in the case of a left turn) in substantially the same time from the start of walking.

  In addition, since the points P1 to P4 are also highlighted in the position display column 506, it is easy for a leader or the like to grasp the actual position of the pedestrian.

  The upper image data in FIG. 38B is the image data of the turning portion from the points P2 to P3, and the lower image data in FIG. 38B is the image data of the turning portion from the points P3 to P2. From the position of the position icon 507, it can be seen that the walking speed of the pedestrian is faster when turning left than when turning right. Therefore, the time required for this pedestrian to reach point P3 from point P2 (in the case of a right turn) is longer than the time required to reach point P2 from point P3 (in the case of a left turn). In such a case, the display control unit 32 stops the left turn image data when the left turn pedestrian reaches the point P2, and reproduces the left turn image data when the right turn pedestrian reaches the point P3. Resume. By doing so, the leader or the like can compare the straight portions of the right turn and the left turn, and can compare the turning portions of the right turn and the left turn.

  FIG. 39 is an example of a flowchart illustrating a procedure in which the display control unit 32 displays image data. The process of FIG. 39 starts with the reproduction of the image data.

  The display control unit 32 determines whether or not the pedestrian has reached the point P2 based on the right-turn image data (S10).

  When the determination in step S10 is No, the display control unit 32 determines whether or not the pedestrian has reached the point P3 with the left turn image data (S20). If the determination in step S20 is No, the process returns to step S10. The determinations in steps S10 and S20 are in no particular order.

  When the determination in step S10 is Yes, the display control unit 32 stops the right turn image data until the pedestrian reaches the point P3 with the left turn image data (S30). That is, the image data of the right turn is stopped until the position θ ′ of the left turn pedestrian reaches P3, and the image data of the right turn is restarted when the position θ ′ reaches P3. The left turn image data is displayed without stopping.

  When the determination in step S20 is Yes, the display control unit 32 stops the left turn image data until the pedestrian reaches the point P2 with the right turn image data (S40). That is, the image data of the left turn is stopped until the position θ of the right turn pedestrian reaches P2, and the image data of the left turn is restarted when the position θ reaches P2. Right-turn image data is displayed without stopping.

  When the display control unit 32 performs this process for each point, the leader or the like can compare the straight portions of the right turn and the left turn, and can compare the turning portions of the right turn and the left turn.

Note that the display control unit 32 may shorten the reproduction time or extend the reproduction time instead of stopping the reproduction of the image data as shown in FIG. The elapsed time until the pedestrian passes each point by the right turn and the left turn is known by the image rotation unit 34. For example, in the case of right turn, the elapsed time of P1 to P2 is Tr, and in the case of left turn, the elapsed time of P4 to P3 is Tl. When Tr> Tl, the display control unit 32 shortens the reproduction time of the right-turn image data.
Reduced playback time = original playback time x (Tl / Tr)
The display control unit 32 reproduces the image by increasing the frame rate at the time of capturing the right turn image data by the reciprocal of Tl / Tr or by thinning the right turn image data so that the number of frames becomes Tl / Tr. Reduce time.

Conversely, the display control unit 32 may extend the reproduction time of the left-turn image data.
Extended playback time = original playback time x (Tr / Tl)
The display control unit 32 extends the reproduction time by reducing the frame rate at the time of image data capture by the reciprocal of Tr / Tl or interpolating so that the number of frames of the image data is multiplied by Tr / Tl. To do.

  By such processing, the leader or the like can compare the straight portions of the right turn and the left turn, and can compare the turning portions of the right turn and the left turn.

<Summary>
As described above, in the image display system 100 of the present embodiment, since a pedestrian turning around the omnidirectional image capturing device 60 or turning left and right is imaged, various walking motions of the pedestrian can be imaged. become. There is no need to move the camera or use motion capture. However, this is not to deny that a pedestrian wearing a marker is imaged. Further, since the acceleration of the pedestrian and the like are displayed on the same time axis as the image data, the instructor and the like can evaluate the walking motion using the image data and the measurement data in a complementary manner. In addition, since one of the right turn and left turn image data is reversed left and right, it is easy for a leader or the like to compare.

<Other configuration examples>
The configuration example in FIG. 2 is merely an example, and the image display system 100 may have the following configuration. FIG. 40A shows another example of a schematic configuration diagram of the image display system 100. In FIG. 40A, the server 10 has a display 108. In this configuration, the server 10 has the function of the display device 30 in FIG. 5, and the server 10 not only synchronizes the time axis but also rotates the omnidirectional image and displays measurement data and image data.

  In FIG. 40B, the server 10 and the display device 30 are eliminated. In this configuration, the acceleration measuring device 50 has the functions of the server 10 and the display device 30 of FIG.

  FIG. 41 is a sequence diagram in which the acceleration measuring device 50 displays measurement data and image data in the configuration as shown in FIG.

  S1: When starting measurement of walking motion, a leader or a pedestrian inputs an operation for starting measurement of walking motion to the acceleration measuring device 50. The acceleration measuring device 50 receives an operation of a leader or a pedestrian.

  S2: The acceleration measuring device 50 transmits an imaging start request to the omnidirectional imaging device 60.

  S3, S4: The acceleration measuring device 50 starts measuring acceleration, and the omnidirectional image capturing device 60 starts capturing a pedestrian.

  S5: When the walking is finished, the leader or the pedestrian inputs an operation to finish the measurement of the walking motion to the acceleration measuring device 50. The acceleration measuring device 50 receives an operation of a leader or a pedestrian.

  S6: The acceleration measuring device 50 transmits an imaging end request to the omnidirectional image capturing device 60.

  S7: The omnidirectional image capturing device 60 ends the image capturing and transmits the image data to the acceleration measuring device 50.

  S8: The acceleration measuring device 50 processes the measurement data as described in the server 10.

  S9: Further, the acceleration measuring device 50 synchronizes the time axes of the measurement data and the image data as performed by the server 10.

  S10: The acceleration measuring device 50 displays the measurement data and the image data on the display 215 of its own device.

  S11: Alternatively, the acceleration measuring device 50 may transmit the measurement data and the image data to a terminal such as a leader (such as the server 10).

  S12: Thereby, a terminal such as a leader displays measurement data and image data.

  Therefore, according to the configuration shown in FIG. 40B, the system configuration can be simplified. A pedestrian walks while wearing the acceleration measuring device 50. After the end of walking, the pedestrian can receive guidance from the instructor by passing the acceleration measuring device 50 to the instructor.

  In addition, even when there are a plurality of pedestrians, the processing when the pedestrian is approaching the end can be applied to the processing when there is only one pedestrian.

<When server 10 determines display range>
In the configuration of FIG. 2, the server 10 may determine the display range. FIG. 42 shows a functional configuration diagram in this case. In the description of FIG. 42, differences from FIG. 2 will be mainly described. In FIG. 42, the server 10 has a moving object detection unit 35. The server 10 detects the moving object and transmits the center O of the circumscribed rectangle 601 or the display range 513 (position information) to the display device 30 in association with each frame. Therefore, the display device 30 does not need to detect moving objects, and the load on the display device 30 can be reduced.

  Further, the server 10 may further include an image rotation unit 34. Accordingly, the server 10 can trim the display range 513 from the omnidirectional image and transmit it to the display device 30. In this case, the communication time between the server 10 and the display device 30 can be shortened, and the memory of the display device 30 is hardly pressed.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, in the present embodiment, the measurement data and the image data are displayed after the measurement data is measured and the image data is captured. However, the measurement data and the image data may be displayed in real time. In this case, the displayed measurement data and image data are taken at almost the same time.

  Further, the configuration example such as FIG. 5 shown in the above embodiment is mainly for the purpose of facilitating understanding of the processing of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50. It is divided according to various functions. For example, a plurality of servers 10 may exist, and a plurality of servers 10 may cooperate to perform the processing of this embodiment. Further, the storage unit 1000 of the server 10 does not need to be included in the server 10, and the storage unit 1000 may be in a location where the server 10 can read and write data in the storage unit 1000.

  In FIG. 6, the functions of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 have been described by dividing them into several processing units. However, the present invention is not limited by the way of dividing or the name of each processing unit. The processing of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 can be divided into more processing units depending on the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  The omnidirectional imaging device 60 is an example of a wide-angle imaging device, the communication unit 11 is an example of an acquisition unit and a transmission unit, the display control unit 32 is an example of a display control unit, and the moving object detection unit 35 is It is an example of a moving body detection means, and the image rotation part 34 is an example of an image cut-out means. The left / right reversing unit 36 is an example of a left / right reversing unit.

10 server 10
12 Image adjustment unit 30 Display device 30
32 Display Control Unit 33 Operation Input Accepting Unit 34 Image Rotating Unit 35 Moving Object Detection Unit 50 Acceleration Measuring Device 60 Global Image Imaging Device 100 Image Display System

JP 2010-172394 A

Claims (18)

An image display system having one or more information processing devices and a display device,
Acquisition means for acquiring image data captured by a wide-angle imaging device capable of imaging the periphery in the horizontal direction;
Display control means for causing the display device to display a moving object shown in the image data;
An image display system. The wide-angle imaging device images the moving object that travels around the wide-angle imaging device while changing the direction or changing the direction,
The image display system according to claim 1, wherein the display control unit displays the moving object on the display device. Moving object detection means for detecting the moving object from the image data;
Image cutting means for cutting out an image of a predetermined range from the image data with the moving body detected by the moving body detection means as the center in the horizontal direction;
3. The image display system according to claim 1, wherein the display control unit displays the image of the predetermined range on the display device to display the moving object at substantially the center of the display device.   The image display system according to claim 3, wherein the moving object detection unit calculates a difference between a plurality of the image data having different captured times, and determines that the moving object exists in a pixel in which the difference is detected. The moving object detection means performs a process of creating a pixel range by tracing a pixel in contact with the pixel in which the difference is detected,
The image display system according to claim 4, wherein when there are a plurality of the pixel ranges in the image data, it is determined that a moving object exists in the largest pixel range. The image data is imaged around 360 ° in the horizontal direction,
The moving object detection means generates first image data and second image data having different horizontal cutting start positions from the one image data,
The image display system according to claim 5, wherein the pixel range is created from each of the first image data and the second image data, and the moving object is detected based on the pixel range having the largest area. The image data is imaged around 360 ° in the horizontal direction, and is cut out at a predetermined cutting start position in the horizontal direction.
The moving object detection means generates the image data by changing the cutting start position in the horizontal direction when the moving object is detected from an end of the image data with less than a threshold value.
The image display system according to claim 5, wherein the pixel range is created from the image data in which the cut start position is changed, and the moving object is detected based on the pixel range.   6. The moving object detection means calculates a moving speed of the moving object based on the position of the moving object detected in the past image data, and estimates the position of the moving object in the image data based on the moving speed. The image display system described in 1. When the number of moving objects detected from the image data due to overlap of the pixel range is less than the number of moving objects detected in the past image data,
The image display system according to claim 8, wherein the moving object detection unit estimates the position of the moving object in the image data based on the moving speed.   The image display system according to claim 3, wherein the moving object detection unit detects the moving object after reducing the image data. The wide-angle imaging device images the moving body that travels in the first turning direction and the moving body that travels in the second turning direction around the wide-angle imaging device,
Left-right reversing means for performing left-right reversal processing on either the image data obtained by imaging the moving object in the first turning direction or the image data obtained by photographing the moving object in the second turning direction,
The image display system according to any one of claims 1 to 10, wherein the display control unit causes the display device to display image data that is horizontally reversed and image data that is not horizontally reversed.   The image display system according to claim 11, wherein the display control unit further displays a position of the moving body in a path along which the moving body has traveled together with the image data.   The display control means is configured such that the position of the moving body that has traveled in the turning direction in which the left-right reversing means has performed the left-right reversing process on the image data is the position in the turning direction in which the left-right reversing process has not been performed. The image display system according to claim 12, wherein the image display system displays the image data together with the image data. The route is preliminarily divided according to the shape of the portion of the route,
The display control means includes image data for moving the portion in the first turning direction;
The part of the same shape as the part in which the moving body moves in the first turning direction is displayed in synchronization with the image data in which the moving body moves in the second turning direction. The image display system described. An image display system having one or more information processing devices,
Acquisition means for acquiring image data captured by a wide-angle imaging device capable of imaging the periphery in the horizontal direction;
Moving object detection means for detecting a moving object reflected in the image data;
Transmitting means for transmitting the image data together with the position information of the moving object to a display device;
An image display system. A display device that displays image data captured by a wide-angle imaging device capable of imaging the surroundings in the horizontal direction,
Obtaining means for obtaining the image data;
Display control means for displaying a moving object shown in the image data;
A display device. Information processing device
Acquisition means for acquiring image data captured by a wide-angle imaging device capable of imaging the periphery in the horizontal direction;
Moving object detection means for detecting a moving object reflected in the image data;
The program for functioning as a transmission means which transmits the said image data to a display apparatus with the positional information on the said moving body. A display device that displays image data captured by a wide-angle imaging device capable of imaging the periphery in the horizontal direction,
Obtaining means for obtaining the image data;
A program for functioning as display control means for displaying a moving object shown in the image data.