JP2017151281A - Image display system, information processing device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display system allowing a progress motion of a moving object to be easily evaluated.SOLUTION: An image display system 100 includes: an information processing device 10; a wide-angle imaging device 60 capable of imaging the periphery in the horizontal direction; and a measuring device 50 for measuring the information allowing the evaluation of a progress motion of a moving object. The image display system includes: acquiring means 11 for acquiring image data taken by the wide-angle imaging device and measurement data measured by the measuring device; associating means 12 for associating the image data with the measurement data; and displaying means 108 for displaying the associated image data and measurement data on a display device 30.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image display system, an information processing apparatus, and a program.

  It is known that walking by people helps to promote health. In addition, when walking, it is preferable for health to walk with a correct posture, and it is difficult to fall. Since it is difficult for the person to evaluate the posture at the time of walking, there are cases in which an instructor or the like related to walking objectively analyzes walking motion or tries to guide the analysis result to the pedestrian.

  A motion capture system is known as a system for measuring the posture and motion of a person or the like. In a motion capture system, a camera can image a person with a marker attached to the joint position of the body, etc., and track the movement of the marker with a tracker, so that the computer can analyze the walking motion of a person .

  However, the motion capture system requires a large number of markers in order to capture the movement of a person or the like from various angles, and there is a tendency that it takes time and cost to prepare for imaging of walking motion and to analyze after imaging. In addition, there is a disadvantage that walking motion can be measured only in a special environment such as a laboratory.

  For such inconvenience, a technique has been devised that can analyze a walking motion with a relatively simple configuration using a captured image of a pedestrian with a plurality of marks attached to a predetermined part of the body (for example, Patent Documents). 1).

  However, the technique disclosed in Patent Document 1 has a problem that walking motions that can be imaged are limited. This is because the position and orientation of the camera are fixed while people and the like move gradually by walking. When the camera is fixed, for example, only a walking motion of a pedestrian who walks linearly can be captured, and it is difficult to capture a walking motion of a pedestrian other than a linear walk.

  If there is a system that changes the direction and position of the camera according to the person's walking, the walking motion that can be imaged can be expanded, but considering the cost and space for that, it is easy to realize as a widely used system Not.

  Further, in Patent Document 1, a parameter value for representing the walking motion is calculated based on the coordinates of the mark. However, even if the walking motion can be evaluated using the parameter value, it is difficult to support the basis for the evaluation. There is.

  In view of the above problems, an object of the present invention is to provide an image display system that makes it easy to evaluate the progress of a moving object.

  In view of the above problems, the present invention is an image display system including an information processing device, a wide-angle imaging device capable of imaging the surroundings in the horizontal direction, and a measurement device that measures information capable of evaluating a traveling operation of a traveling object, Acquisition means for acquiring image data captured by the wide-angle imaging device and measurement data measured by the measurement device, association means for associating the image data with the measurement data, the associated image data and the measurement Display means for displaying data on a display device.

  It is possible to provide an image display system that makes it easy to evaluate the progress of a moving object.

It is an example of the figure explaining schematic operation | movement of an image display system. 1 is an example of a schematic configuration diagram of an image display system. It is an example of the hardware block diagram of a server. It is an example of the hardware block diagram of an acceleration measuring device. It is an example of the hardware block diagram of an omnidirectional image pick-up device. It is an example of a functional block diagram of a server, a display device, an omnidirectional image capturing device, and an acceleration measuring device. It is an example of the figure explaining how a pedestrian walks the circumference | surroundings of an omnidirectional image imaging device. It is an example of the sequence diagram explaining the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is an example of the sequence diagram explaining the modification of the whole operation | movement of an image display system. It is a figure which shows an example of the image data and measurement data which the display apparatus displayed on the display. It is an example of the flowchart figure which shows the procedure in which an image rotation part specifies the range which the pedestrian is reflected, and displays it on an image data column. It is a figure which shows typically the position of the foot of a pedestrian in the top view of an omnidirectional image. It is an example of the flowchart figure which shows the procedure in which an image rotation part specifies the range which the pedestrian is reflected, and displays it on an image data column. It is an example of the figure which illustrates face recognition typically. Another example of a schematic block diagram of an image display system is shown. FIG. 19 is a sequence diagram in which the acceleration measuring device displays measurement data and image data in the configuration as shown in FIG.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

<Outline of image processing system>
FIG. 1 is an example of a diagram illustrating a schematic operation of the image display system 100 according to the present embodiment. An omnidirectional image capturing device 60 is installed, and the omnidirectional image capturing device 60 captures a pedestrian walking around the omnidirectional image capturing device 60. The pedestrian 200 wears the acceleration measuring device 50 on the body, and the acceleration measuring device 50 measures the acceleration generated in the pedestrian 200 being imaged.

  (1) The omnidirectional image capturing device 60 transmits image data to the server 10, and the acceleration measuring device 50 transmits acceleration measurement data to the server 10.

  (2) The imaging start time of image data and the measurement start time of measurement data are synchronized, and the imaging end time of image data and the measurement end time of measurement data are synchronized. For this reason, the server 10 can synchronize the time axes of image data and measurement data.

  (3) The server 10 transmits the image data and measurement data whose time axes are synchronized to the display device 30. Thereby, the display device 30 can display the image data and the measurement data on the same time axis.

  According to such a configuration, first, the omnidirectional image capturing device 60 is used, and the pedestrian 200 walking around the pedestrian 200 is imaged. It becomes possible to turn left or take an image of the pedestrian 200. It is known that the pedestrian 200 easily loses its center of gravity when turning or turning left and right, and the ability to capture an image of the pedestrian 200 turning or turning left and right is effective for analysis of walking motion. Further, since the omnidirectional image capturing device 60 can always capture the side surface of the pedestrian 200 without moving the camera, it can capture how the body of the pedestrian 200 shakes in the front-rear direction. An instructor or the like who walks can check the walking motion of the pedestrian 200 with an image, evaluate the walking motion, and guide improvement points.

  In addition, since the measurement data in which the acceleration of the pedestrian 200 is measured is displayed on the same time axis as the image data, when the instructor or the like finds a walking motion that should be improved, the walking motion is measured to be not good. Can be supported by data. On the other hand, by comparing the measurement data prepared in advance, which is considered good, with the measurement data of the pedestrian (subject), the instructor can find undesirable measurement data. Since the unfavorable measurement data is associated with the image data, the instructor or the like can specify the walking motion to be improved from the image data. Therefore, the walking motion can be evaluated by using the measurement data and the image data in a complementary manner.

<Terminology>
A traveling object refers to an object that travels forward, backward, laterally, or the like. Alternatively, it is an object that travels with the movement of the center of gravity. Specific examples include humans, animals, robots, and the like, but in the following description, humans (pedestrians) will be described as an example for explanation. Walking includes running. You may also walk backwards. In addition, a wheelchair on which a person or the like gets on, a person wearing a prosthetic leg, a bicycle, a stilt, a tricycle, or the like may be captured.

  The information that can evaluate the progress of the progressing object refers to information that varies with the progress of the progressing object. Alternatively, it may be information that varies with progress when attached to a target that travels. Or you may say that progress operation | movement is the information which can evaluate quality etc. Specifically, it is acceleration, angular velocity, position or orientation.

  Correspondence between image data and measurement data means that when one or more pieces of image data are determined from some image data, one or more pieces of measurement data are determined. Alternatively, image data and measurement data measured at substantially the same time are specified, and when one of time, image data, or measurement data is determined, the rest is determined.

  A leader refers to a person who can confirm a walking motion of a pedestrian with a video, evaluate the walking motion, and plan improvement points. Specific examples include holders of qualifications such as public health nurses, clinical laboratory technicians, and prosthetic limb orthoses. Moreover, even if it is not a person who can evaluate a walking motion and can plan an improvement point etc., operation of the image display system 100 is possible, and a leader may be called an operator, a user, or a user.

<System configuration example>
FIG. 2 shows an example of a schematic configuration diagram of the image display system 100 of the present embodiment. The image display system 100 includes a server 10, an acceleration measuring device 50, an omnidirectional image capturing device 60, and a display device 30 that can communicate with each other via a network N.

  The network N is constructed by a LAN constructed in a facility where the image display system 100 is arranged, a provider network of a provider that connects the LAN to the Internet, a line provided by a circuit provider, and the like. When the network N has a plurality of LANs, the network N is called a WAN or the Internet. The network N may be constructed by either wired or wireless, and wired and wireless may be combined. When the display device 30 is directly connected to a line telephone network or a mobile phone network, it can be connected to a provider network without going through a LAN.

  The omnidirectional image capturing device 60 is an image capturing device (camera) that captures a 360-degree landscape called an omnidirectional image or an omnidirectional image (hereinafter simply referred to as an omnidirectional image or an omnidirectional image). By providing an optical system to be described later, the omnidirectional image capturing device 60 can obtain an omnidirectional image by one imaging operation. Since an image of 360 degrees around the omnidirectional image is picked up with almost no blind spot, an instructor or the like can selectively display an arbitrary region on a display or the like for browsing. In the present embodiment, it is not necessary to have an imaging range of 360 degrees in the latitude direction (up and down direction of the image). This is because pedestrians are not shown in the ceiling and floor areas. Further, it may not be possible to capture an image in the entire range of 360 degrees in the longitude direction (the horizontal direction of the image). This is because it may not be necessary to turn the entire omnidirectional image capturing device 60 in order for the instructor to analyze the walking motion of the pedestrian. Further, as the omnidirectional image capturing device 60, an image capturing device that obtains an omnidirectional image by connecting a plurality of images captured by an image capturing device that captures a rectangular range of a certain angle of view may be used.

  The acceleration measuring device 50 is a device that measures the acceleration of a pedestrian. The acceleration is preferably measured on each of the three axes of vertical acceleration, traveling acceleration, and lateral acceleration, but may be acceleration in any one direction. Although there is a dedicated sensor for measuring acceleration as the acceleration measuring device 50, the acceleration measuring device 50 may be a device having an acceleration sensor. Specific examples include smartphones, mobile phones, tablet terminals, notebook PCs, wearable PCs (mounted as head mounted displays, sunglasses, watches, etc.), pedometers, and the like, but are not limited thereto.

Further, the acceleration measuring device 50 may have a function of measuring angular velocity, position, orientation, etc. in addition to acceleration. The sensor that measures the angular velocity is called a gyro sensor, and the position is provided to the acceleration measuring device 50 by a GPS satellite, IMES, or a beacon. A sensor that detects the direction of east, west, north, and south, for example, with reference to the north is called an orientation sensor. Any of these can be used for the evaluation of the walking motion as well as the acceleration. In the present embodiment, the acceleration measuring device 50 is described as measuring acceleration for the sake of simplicity, but the acceleration measuring device 50 can transmit measurement data used for evaluating these walking motions to the server 10.
The server 10 is an information processing device that receives and processes image data from the omnidirectional image capturing device 60, or receives measurement data and synchronizes the image data with the time axis. The function of the server 10 can be shared by the acceleration measuring device 50 and the display device 30.

  The display device 30 is an information processing device used as a terminal device that allows an instructor or the like to display image data and measurement data. Specifically, for example, PC (Personal Computer), smart phone, tablet device, mobile phone, car navigation terminal, wearable computer (head mounted display, sunglasses, wristwatch, etc.), camera, electronic blackboard, projector, game machine, or MFP (Multifunction Peripheral / Printer / Product) etc. are mentioned.

<Hardware configuration example>
FIG. 3 is an example of a hardware configuration diagram of the server 10. Note that the hardware configuration of the server 10 shown in the figure does not need to be housed in a single casing or provided as a single device, and represents a hardware element that is preferably provided in the server 10. . The configuration of the server 10 may be determined by cloud computing in which resources are appropriately allocated according to a load or the like.

  The server 10 includes a CPU 101, ROM 102, RAM 103, HDD 105, media drive 107, display 108, network I / F 109, keyboard 111, mouse 112, and optical drive 114 connected to the bus 110. The CPU 101 executes an image processing program stored in the HD 104 and controls the overall operation of the server 10. The ROM 102 stores a program used for driving the CPU 101 such as an IPL (Initial Program Reader). A RAM 103 is a main storage device used as a work area for the CPU 101. The HD 104 is a storage device equipped with a nonvolatile memory.

  An HDD (Hard Disk Drive) 105 controls reading or writing of various data with respect to the HD 104 according to the control of the CPU 101. A display (display device 30) 108 displays various information such as a cursor, menu, window, character, or image. A network I / F 109 is an interface with the network N.

  A keyboard 111 and a mouse 112 are input / output devices. The keyboard 111 includes a plurality of keys for inputting characters, numerical values, various instructions, and the like, and receives inputs from these keys. The mouse 112 receives movement of the mouse pointer, selection and execution of various instructions, selection of a processing target, and the like.

  The media drive 107 controls reading or writing (storage) of data with respect to the recording medium 106 such as a flash memory. The optical drive 114 controls reading or writing of various data with respect to a CD (Compact Disc) 113 as an example of a removable recording medium.

  The image processing program may be installed in an installable or executable format, recorded on a computer-readable recording medium such as the recording medium 106 or the CD 113, and distributed. Alternatively, the image processing program may be distributed in a form downloaded from any server 10 type information processing apparatus.

  It should be noted that the hardware configuration of the display device 30 is the same as or different from that in FIG.

  FIG. 4 shows an example of a hardware configuration diagram of the acceleration measuring device 50. In FIG. 4, a smartphone or a tablet device is assumed as the acceleration measuring device 50, but the hardware configuration of the acceleration measuring device 50 is not limited to this.

  The acceleration measuring apparatus 50 includes a CPU 201, a ROM 202, a RAM 203, a flash memory 204, a CMOS sensor 205, an acceleration / orientation sensor 206, a gyro sensor 207, a media drive 208, an audio input unit 211, and an audio output unit that are connected to the bus 210. 212, a communication device 213, a GPS receiver, a display 215, and a touch panel 216. The bus 210 is an address bus, a data bus, or the like for electrically connecting these units.

  The CPU 201 controls the overall operation of the display device 30 by executing a program stored in the flash memory 204. The ROM 202 stores IPL and basic input / output programs. A RAM 203 is a main storage device used as a work area when the CPU 201 executes a program. The flash memory 204 is a nonvolatile storage device that stores programs executed by the display device 30 and various data. The program is, for example, an operating system and a device program executed by the omnidirectional image capturing device 60.

  The CMOS sensor 205 is an image sensor that captures an image of a subject under the control of the CPU 201 and obtains image data. A CCD sensor may be used instead of the CMOS sensor 205. The acceleration / orientation sensor 206 has a function as an electronic magnetic compass for detecting geomagnetism, a function for measuring acceleration in three axial directions, and the like. The gyro sensor 207 detects an angular velocity when the acceleration measuring device 50 rotates with respect to the x axis, the y axis, or the z axis. The rotation angle of each axis is called a yaw angle, a pitch angle, and a roll angle.

  The media drive 208 controls reading or writing (storage) of data with respect to a recording medium 209 such as a flash memory. According to the control of the media drive 208, the recording medium 209 from which already recorded data is read or newly written and stored is detachable.

  The voice input unit 211 is a microphone that converts voice into a voice signal. The audio output unit 212 is a speaker that converts an audio signal into audio. The communication device 213 communicates with the nearest wireless station device 9 by a wireless communication signal using the antenna 213a. Alternatively, the communication device 213 may be a LAN card connected to the LAN. The GPS receiving unit 214 detects position information (latitude, longitude, and altitude) of the display device 30 by an IMES (Indoor MEssaging System) as a GPS satellite or indoor GPS.

  The display 215 displays an omnidirectional image under the control of the CPU 201 and displays various menus, icons, and the like for the user to operate the display device 30. The touch panel 216 is integrally superimposed on the display 215 and detects a touch position (coordinates) on the display 215 with respect to a touch with a finger or a touch pen.

  The device program may be recorded in a computer-readable recording medium such as the recording medium 209 and distributed as a file in an installable or executable format. Alternatively, the device program may be distributed in a form downloaded from any server 10 type information processing device.

  FIG. 5 is an example of a hardware configuration diagram of the omnidirectional image capturing apparatus 60. In the following, the omnidirectional image capturing device 60 is an omnidirectional image capturing device using two image capturing elements, but the number of image capturing elements may be three or more. In addition, it is not always necessary to use an apparatus dedicated to omnidirectional imaging. By attaching a retrofit omnidirectional imaging unit to a normal digital camera or smartphone, it has substantially the same function as the omnidirectional imaging apparatus 60. It may be.

  As shown in FIG. 5, the omnidirectional image capturing apparatus 60 includes an image capturing unit 301, an image processing unit 304, an image capturing control unit 305, a microphone 308, a sound processing unit 309, an audio output unit 319, a CPU 311, a ROM 312, An SRAM 313, a DRAM 314, an operation unit 315, a network I / F 316, a communication device 317, and an antenna 317a are provided.

  Among these, the imaging unit 301 includes wide-angle lenses (so-called fish-eye lenses) 302a and 302b each having an angle of view of 180 ° or more for forming a hemispherical image, and two imaging units provided corresponding to the wide-angle lenses. Elements 303a and 303b are provided. The image pickup elements 303a and 303b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by a fisheye lens into image data of an electrical signal and output the image data. A timing generation circuit for generating a vertical synchronization signal, a pixel clock, and the like, and a register group in which various commands and parameters necessary for the operation of the image sensor are set.

  The imaging elements 303a and 303b of the imaging unit 301 are each connected to the image processing unit 304 by a parallel I / F bus. On the other hand, the imaging elements 303 a and 303 b of the imaging unit 301 are connected to a serial I / F bus (I2C bus or the like) separately from the imaging control unit 305. The image processing unit 304 and the imaging control unit 305 are connected to the CPU 311 via the bus 310. Further, ROM 312, SRAM 313, DRAM 314, operation unit 315, network I / F 316, communication device 317, electronic compass 318, and the like are also connected to the bus 310.

  The image processing unit 304 takes in the image data output from the image sensors 303a and 303b through the parallel I / F bus, performs predetermined processing on the respective image data, and then combines these image data. Creates equirectangular image data.

  The imaging control unit 305 generally sets commands and the like in the register groups of the imaging elements 303a and 303b using the I2C bus with the imaging control unit 305 as a master device and the imaging elements 303a and 303b as slave devices. Necessary commands and the like are received from the CPU 311. The imaging control unit 305 also takes in the status data of the register groups of the imaging elements 303a and 303b using the I2C bus and sends them to the CPU 311.

  The imaging control unit 305 instructs the imaging elements 303a and 303b to output image data at the timing when the shutter button of the operation unit 315 is pressed. Some omnidirectional image capturing devices 60 may have a preview display function by a display or a function corresponding to a moving image display. In this case, output of image data from the image sensors 303a and 303b is continuously performed at a predetermined frame rate (frame / minute).

  Further, as will be described later, the imaging control unit 305 also functions as a synchronization control unit that synchronizes the output timing of the image data of the imaging elements 303a and 303b in cooperation with the CPU 311. In the present embodiment, the omnidirectional image capturing device 60 is not provided with a display unit, but may be provided with a display unit.

  The microphone 308 converts sound into sound (signal) data. The sound processing unit 309 takes in sound data output from the microphone 308 through the I / F bus and performs predetermined processing on the sound data. The audio output unit 319 is a speaker that converts an audio signal into audio.

  The CPU 311 controls the overall operation of the omnidirectional image capturing apparatus 60 and executes necessary processes. The ROM 312 stores various programs for the CPU 311. The SRAM 313 and the DRAM 314 are work memories, and store programs executed by the CPU 311 and data being processed. In particular, the DRAM 314 stores image data being processed by the image processing unit 304 and image data created by a processed equirectangular projection.

  The operation unit 315 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that has both display and operation functions, and the like. The user inputs various imaging modes, imaging conditions, and the like by operating the operation buttons.

  The network I / F 316 is a general term for an interface circuit (USB I / F or the like) with an external medium such as an SD card or a personal computer. The network I / F 316 may be a network interface regardless of wireless or wired. Data of the image created by the equirectangular projection stored in the DRAM 314 is recorded on an external medium via the network I / F 316, or the network I / F 316 which becomes a network I / F as necessary. Or transmitted to an external device.

  The communication device 317 communicates with an external device via a wireless technology such as WiFi (wireless fidelity), NFC, or LTE (Long Term Evolution) via an antenna 317a provided in the omnidirectional image capturing device 60. The communication device 317 can also transmit image data created by equirectangular projection to an external device.

  The electronic compass 318 calculates the azimuth and tilt (Roll rotation angle) of the omnidirectional imaging device 60 from the earth's magnetism, and outputs azimuth / tilt information. This azimuth / tilt information is an example of related information (metadata) along Exif, and is used for image processing such as image correction of a captured image. The related information includes each data of the image capturing date and time and the data capacity of the image data.

<Functional configuration>
Next, functional configurations of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 will be described with reference to FIG. FIG. 6 is an example of a functional block diagram of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

<< Functional configuration of server >>
The server 10 includes a communication unit 11, an image adjustment unit 12, an acceleration data processing unit 13, an operation input receiving unit 14, a synchronization instruction unit 15, and a storage / reading unit 19. Each of these units included in the server 10 is realized by any one of the components illustrated in FIG. 3 operating according to an instruction from the CPU 101 according to the image processing program 1010 expanded from the HD 104 onto the RAM 103. Function or means to be functioned.

  In addition, the server 10 includes a storage unit 1000 constructed by one or more of the HD 104, the ROM 102, and the RAM 103 illustrated in FIG. The storage unit 1000 stores an image processing program 1010, a measurement data DB 1001, and an image data DB 1002. The measurement data DB 1001 is a storage unit that stores measurement data transmitted from the acceleration measurement device 50. The image data DB 1002 is a storage unit that stores image data transmitted from the omnidirectional image capturing device 60.

  The communication unit 11 of the server 10 is realized by a command from the CPU 101 illustrated in FIG. 3 and a network I / F 109, and the like, and via the network N, the acceleration measuring device 50, the omnidirectional image capturing device 60, and Various data are transmitted to and received from the display device 30. In the following description, even when the server 10 communicates with the function of the communication unit 11, the description “communication via the communication unit 11” may be omitted. The same applies to the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

  The image adjustment unit 12 is realized by a command or the like from the CPU 101 shown in FIG. 3, and synchronizes the time axis of the image data acquired from the omnidirectional image capturing device 60 and the measurement data acquired from the acceleration measuring device 50. .

  The acceleration data processing unit 13 is realized by a command or the like from the CPU 101 shown in FIG. 3, and converts discrete measurement data into a graph with respect to the time axis by smoothly combining them.

  The operation input receiving unit 14 is realized by a command from the CPU 101 illustrated in FIG. 3, a keyboard 111, a mouse 112, and the like, and receives various operations and inputs to the display device 30 by a leader or the like.

  The synchronization instructing unit 15 is realized by an instruction from the CPU 101 shown in FIG. 3 and performs processing related to synchronization of measurement data and image data. Specifically, the omnidirectional image capturing device 60 and the acceleration measuring device 50 are requested to start measurement and start imaging at the same time, or the omnidirectional image capturing device 60 and the acceleration measuring device to end measurement and image capturing at the same time. Or request to 50. Also, the absolute time is provided to the omnidirectional image capturing device 60 and the acceleration measuring device 50.

  The storage / reading unit 19 is realized by an instruction from the CPU 101 and the HDD 105 shown in FIG. 3, and stores various data in the storage unit 1000 or reads various data stored in the storage unit 1000. I do. In the following description, even when the server 10 accesses the storage unit 1000, the description “read / write via the storage / reading unit 19” may be omitted. The same applies to the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50.

<< Functional configuration of display device 30 >>
The display device 30 includes a communication unit 31, a display control unit 32, an operation input reception unit 33, an image rotation unit 34, and a storage / readout unit 39.

  Each unit included in the display device 30 is realized by any one of the constituent elements illustrated in FIG. 3 operating according to a command from the CPU 101 according to the terminal program 3010 expanded from the HD 104 onto the RAM 103. A function, or a means to be functioned. In addition, the function implement | achieved by the web page (The program described by HTML data, XML data, JavaScript (trademark), etc.) which is not restricted to the terminal program 3010, and a function is included. .

In addition, the display device 30 includes a storage unit 3000 configured by one or more of the HD 104, the ROM 102, and the RAM 103 illustrated in FIG. The storage unit 3000 stores a terminal program 3010. The storage unit 3000 stores measurement data and image data transmitted from the server 10.
(Function of the display device 30)
The communication unit 31 of the display device 30 is realized by a command from the CPU 101 illustrated in FIG. 3 and a network I / F 109 and the like, and transmits and receives various data to and from the server 10 via the network N.

  The display control unit 32 is realized by a command from the CPU 101 shown in FIG. 3 and the like, and displays the image data and measurement data transmitted from the server 10 on the display 108.

  The operation input accepting unit 33 is realized by a command from the CPU 101 shown in FIG. 3, a keyboard 111, a mouse 112, and the like, and accepts various operations and inputs to the display device 30 by an instructor or the like.

  The image rotation unit 34 is realized by a command from the CPU 101 shown in FIG. 3 and the like, and image data that is an omnidirectional image is horizontally displayed so that an area where a pedestrian is shown is displayed on the display 108. Rotate (or trim).

  The storage / reading unit 39 is realized by an instruction from the CPU 101 and the HDD 105 shown in FIG. 3, and stores various data in the storage unit 3000 or reads out various data stored in the storage unit 3000. I do.

<< Functional configuration of acceleration measuring device 50 >>
The acceleration measuring device 50 includes a communication unit 51, an acceleration detection unit 52, and a storage / readout unit 59. These components of the acceleration measuring device 50 are realized by any one of the components shown in FIG. 4 operating according to commands from the CPU 201 according to the device program expanded from the flash memory 204 onto the RAM 203. Function or means to be functioned.

Further, the acceleration measuring apparatus 50 includes a storage unit 5000 constructed by any one or more of the flash memory 204, the ROM 202, and the RAM 203 shown in FIG. The storage unit 5000 stores an apparatus program 5010. The storage unit 5000 stores measurement data such as acceleration measured by the acceleration measuring device 50.
(Function of acceleration measuring device 50)
The communication unit 51 of the acceleration measuring device 50 is realized by a command from the CPU 201 illustrated in FIG. 4 and the communication device 213 and the like, and transmits and receives various data to and from the server 10 via the network N.

  The acceleration detection unit 52 is realized by a command from the CPU 201 shown in FIG. 4, the acceleration / direction sensor 206, and the like, and measures the acceleration generated in the acceleration measuring device 50. Acceleration is measured at regular intervals or irregularly to obtain acceleration that varies with time. Measurement data including time-series acceleration is stored in the storage unit 5000.

  The storage / reading unit 59 is realized by the instruction from the CPU 201 illustrated in FIG. 4, the flash memory 204, the ROM 202, the RAM 203, and the like, and stores various data in the storage unit 5000 or stored in the storage unit 5000. Performs processing to read various data.

<< Functional configuration of omnidirectional imaging device 60 >>
The omnidirectional image capturing device 60 includes a communication unit 61, an image capturing unit 62, and a storage / reading unit 69. Each of these components included in the omnidirectional image capturing device 60 operates according to an instruction from the CPU 311 according to the image capturing program developed from the ROM 312 to the SRAM 313 or the DRAM 314. A function realized by or a means to be functioned.

  Further, the omnidirectional image capturing device 60 includes a storage unit 6000 constructed by one or more of the ROM 312, the SRAM 313, and the DRAM 314 shown in FIG. 5. The storage unit 6000 stores image data of an omnidirectional image and an imaging program 6010.

(Function of the omnidirectional imaging device 60)
The communication unit 61 of the omnidirectional image capturing apparatus 60 is realized by a command from the CPU 101 illustrated in FIG. 5 and a network I / F 316 or the like, and transmits and receives various data to and from the server 10 via the network N.

  The imaging unit 62 is realized by a command from the CPU 101 shown in FIG. 5, an imaging unit 301, an image processing unit 304, and the like, and captures an omnidirectional image and generates image data. The spherical image may be a still image or a moving image. In any case, the imaging unit 62 images a walking pedestrian over time. In the case of a moving image, the omnidirectional image is captured at regular intervals or irregularly. In the case of a still image, a plurality of omnidirectional images are captured at time intervals while the pedestrian is walking. Image data including time-series spherical images is stored in the storage unit 6000.

  The storage / reading unit 69 is realized by any one or more of the instruction from the CPU 201 shown in FIG. 5, ROM 312, SRAM 313, and DRAM 314, and stores various data in the storage unit 6000 or storage in the storage unit 6000. A process for reading out various data is performed.

<About walking of pedestrians>
How the pedestrian walks around the omnidirectional image capturing device 60 will be described with reference to FIG. FIG. 7A is an example of a diagram schematically illustrating a pedestrian 200 walking around the omnidirectional image capturing device 60. The pedestrian 200 wears the acceleration measuring device 50 on the waist or the like. The place where the acceleration measuring device 50 is mounted is preferably a place close to the center of gravity of the pedestrian 200. In addition to the waist, it may be attached to the chest, back, torso, etc. A plurality of acceleration measuring devices 50 may be mounted.

  The pedestrian 200 walks while turning around the omnidirectional image capturing device 60. A circular line L may be formed on the walking surface in order to facilitate comparison of walking motions between a plurality of pedestrians or to recognize a route on which the pedestrian 200 walks. In addition, markers with numbers or the like that are continuously spaced at equal intervals may be arranged on the line L. When a leader or the like confirms the image data, the position of the pedestrian 200 with respect to the omnidirectional image capturing device 60 can be easily grasped.

  The pedestrian 200 moves forward with the front (chest) facing in the tangential direction of the circle indicated by the line L. Thereby, it walks so that the circumference | surroundings of the omnidirectional image imaging device 60 may turn. While the pedestrian 200 is walking, the acceleration measuring device 50 measures acceleration. The relative positions of the pedestrian 200 and the acceleration measuring device 50 are fixed, and the traveling direction (front) of the pedestrian 200 is the x direction, the left / right direction is the y direction, and the vertical direction is the z direction.

  Note that the pedestrian 200 does not have to walk the entire circular line L, but may walk more than a distance appropriate for the evaluation of the walking motion. Moreover, the pedestrian 200 may walk the circular line L more than 1 round.

  FIG. 7B is an example of a diagram schematically illustrating a pedestrian 200 walking around the omnidirectional image capturing device 60 along a square. In FIG. 7B, for example, a square line L is formed on the walking surface. When the pedestrian 200 walks along the line L (or on the line), the pedestrian 200 makes a right turn, so that the omnidirectional image pickup device 60 can pick up the pedestrian 200 when making a right turn. Further, if the pedestrian 200 walks in the opposite direction, the omnidirectional image capturing device 60 can capture the pedestrian 200 when turning left.

  The line L has a straight line portion S. Since the pedestrian 200 reciprocates along (or on) the straight line portion S, the omnidirectional image pickup device 60 is a pedestrian 200 walking from behind, a pedestrian 200 performing 180-degree change of direction, And the pedestrian 200 approaching the omnidirectional image pickup device 60 from the front can be picked up.

  Moreover, the path | route which the pedestrian 200 walks may be figure 8 (8-shaped loop, or infinite (infinity shape) etc.), and is not restricted to what was shown in Drawing 7. Line L shows walking motion. Any shape that is preferable for evaluation may be used.

<Operation procedure>
Subsequently, an operation procedure of the image display system 100 will be described with reference to FIGS. FIG. 8 is an example of a sequence diagram illustrating the overall operation of the image display system 100. FIG. 8 is a sequence diagram when the server 10 instructs the acceleration measuring device 50 and the omnidirectional image capturing device 60 to start measurement and start imaging, respectively.

  S1: When starting measurement of walking motion, a leader or the like inputs an operation for starting measurement of walking motion to the server 10. The operation input receiving unit 14 of the server 10 receives an operation of a leader or the like.

  S2: The communication unit 11 of the server 10 transmits a measurement start request to the acceleration measuring device 50.

  S3: The communication unit 51 of the acceleration measuring device 50 receives the measurement start request. By acquiring the measurement start request, the acceleration measuring device 50 outputs a buzzer sound or the like from the sound output unit 212. Thereby, the pedestrian can grasp | ascertain that a walk may be started. The pedestrian may be informed of the start of walking by a voice message such as “Please start walking”, vibration by the vibration function, display of a message on the display 215, blinking of the display 215, or the like.

  S4: The communication unit 11 of the server 10 transmits an imaging start request to the omnidirectional image imaging device 60.

  S5: The communication unit 61 of the omnidirectional image capturing apparatus 60 receives the imaging start request. Upon acquisition of the imaging start request, the omnidirectional image imaging device 60 outputs a buzzer sound or the like from the audio output unit 319. Thereby, the pedestrian can grasp | ascertain that a walk may be started. Other notification modes are the same as in step S3. In addition, since the omnidirectional image capturing apparatus 60 may enter the pedestrian's field of view, the omnidirectional image capturing apparatus 60 may light a patrol lamp or the like. Thereby, not only a pedestrian but the surrounding leaders can grasp that a pedestrian may start walking. Note that only one of the notifications by the buzzer in steps S3 and S5 is sufficient.

  S6, S7: In response to the measurement start request, the acceleration detector 52 of the acceleration measuring device 50 starts measuring the acceleration. In response to the imaging start request, the imaging unit 62 of the omnidirectional image imaging device 60 starts imaging. The acceleration detector 52 of the acceleration measuring device 50 measures the acceleration that occurs while the pedestrian is walking, and the imaging unit 62 of the omnidirectional image capturing device 60 images the walking pedestrian. In this way, the measurement data measurement start and the image data imaging start can be synchronized almost simultaneously by an instruction from the server 10. Note that the order of steps S2 to S7 is random. The server 10 may transmit the measurement data measurement start and the image data imaging start to the acceleration measurement device 50 and the omnidirectional image imaging device 60 at the same time.

  S8: When the leader or the like determines that the pedestrian has sufficiently walked, the leader or the like inputs an operation to end the measurement of the walking motion to the server 10. The operation input receiving unit 14 of the server 10 receives an operation of a leader or the like.

  S9: The communication unit 11 of the server 10 transmits a measurement end request to the acceleration measuring device 50. The communication unit 11 of the acceleration measuring apparatus 50 receives the measurement end request, and the acceleration detection unit 52 ends the acceleration measurement. Note that a pedestrian or the like may be notified of the end as in step S3.

  S10: The communication unit 11 of the server 10 transmits an imaging end request to the omnidirectional imaging device 60. The communication unit 61 of the omnidirectional image imaging device 60 receives the imaging end request, and the imaging unit 62 ends imaging of the omnidirectional image. Note that a pedestrian or the like may be notified of the end as in step S5. The order of steps S9 and S10 is random. The server 10 may transmit the measurement data measurement end and the image data imaging end to the acceleration measuring device 50 and the omnidirectional image capturing device 60 at the same time.

  In this way, by the instruction from the server 10, the measurement data measurement end and the image data imaging end can be synchronized almost simultaneously.

  S11: The communication unit 51 of the acceleration measuring device 50 transmits the measurement data stored in the storage unit 5000 to the server 10.

  S12: The communication unit 61 of the omnidirectional image capturing apparatus 60 transmits the image data stored in the storage unit 6000 to the server 10.

  S13: The communication unit 11 of the server 10 receives the measurement data and the image data, and stores them in the measurement data DB 1001 and the image data DB 1002, respectively. Then, the acceleration data processing unit 13 of the server 10 processes acceleration data that is discrete data into a line graph or the like. Such processing may be performed by the display device 30.

  S14: Next, the image adjustment unit 12 of the server 10 performs a process of synchronizing the time axes of the measurement data and the image data. Although the measurement data and the image data are acquired at substantially the same time, the number of measurement data and the number of image data (frame number) are often different. For this reason, the image adjustment unit 12 synchronizes the time axis by associating the measurement data with the image data. Since the measurement start time and the imaging start time are almost the same, and the measurement end time and the imaging end time are almost the same, the time axis can be synchronized with the ratio of the number of data of the measurement data and the image data. For example, if the number of measurement data is 400 and the number of image data is 200, two frames of image data are associated with one acceleration of the measurement data. Alternatively, the ratio of the measurement time of each measurement data (acceleration) with respect to the measurement time is calculated, the ratio of the imaging time of each image data (frame) with respect to the imaging time is calculated, and the frame with the closest ratio is associated with the acceleration. Good. Alternatively, the measurement data and the image data may be associated with the elapsed time from the start of measurement and the elapsed time from the start of imaging.

  Furthermore, when the absolute time of the measurement start time, the measurement end time, the imaging start time, and the imaging end time is obtained as described later, an overlapping portion of the measurement end time from the measurement start time and the imaging end time from the imaging start time is obtained. Take out and associate the measurement data and image data of the overlapping part by the same ratio calculation.

  S15, S16: The server 10 transmits the measurement data and the image data to the display device 30. Specifically, a leader or the like requests measurement data and image data from the server 10 via the display device 30 or the like. Alternatively, a leader or the like may perform an operation of requesting measurement data and image data from the server 10 and the operation input receiving unit 14 of the server 10 may receive this operation. Alternatively, when the display device 30 inquires about the presence or absence of measurement data and image data by polling or the like and the measurement data and image data are transmitted to the server 10, the server 10 transmits the measurement data and image data to the display device 30 in response to the inquiry. May be.

  S17: The communication unit 31 of the display device 30 receives the measurement data and the image data, and the display control unit 32 displays the measurement data and the image data on the display 108 on the same time axis. Processing for display will be described with reference to FIG.

<< Operation Variation >>
FIG. 9 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 9, differences from FIG. 8 will be described. In FIG. 9, the trigger for starting measurement is different from that in FIG.

  S1: When starting measurement of walking motion, a leader or the like inputs an operation for starting measurement of walking motion to the display device 30. The operation input receiving unit 33 of the display device 30 receives an operation of a leader or the like.

  S1-1: The communication unit 31 of the display device 30 transmits a measurement start request to the server 10. The display device 30 and the server 10 can communicate by HTTP or the like. The measurement start request requests at least one of acceleration measurement start and pedestrian imaging start.

  S8: When the leader or the like determines that the pedestrian has sufficiently walked, the leader or the like inputs an operation to end the measurement of the walking motion to the display device 30. The operation input receiving unit 33 of the display device 30 receives an operation of a leader or the like.

  S8-1: The communication unit 31 of the display device 30 transmits a measurement end request to the server 10. The subsequent processing is the same as in FIG. The measurement end request is a request for at least one of the end of measurement of acceleration and the end of imaging of a pedestrian.

  Therefore, the instructor can request the server 10 to start and end the measurement from the display device 30 without operating the server 10. When a leader or the like is operating the display device 30, it is not necessary to move to the server 10.

<< Operation Variation >>
FIG. 10 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 10, the difference from FIG. 8 will be described. In FIG. 10, the start of measurement is that the pedestrian has started walking. The pedestrian starts walking and ends with the guidance of a leader or the like, for example. Alternatively, the circuit goes around the line from the start position on the line and stops at the start position.

  S1: The acceleration detector 52 of the acceleration measuring device 50 detects that a pedestrian has started walking based on the acceleration. For example, an acceleration exceeding a threshold value was detected in one or more of the x direction, the y direction, and the z direction, or a change in position per unit time obtained by integrating the acceleration twice exceeded the threshold value. The start of walking can be determined by, etc. The determination may be made based on the angular velocity detected by the gyro sensor 207, the position detected by the GPS receiver 214, and the direction detected by the acceleration / direction sensor 206. By detecting the start of walking, the acceleration detector 52 starts measuring acceleration (S6).

  S2: The communication unit 51 of the acceleration measuring device 50 transmits a walking start notification to the server 10. Thereby, the server 10 can transmit an imaging start request to the omnidirectional image capturing device 60. Note that the communication unit 51 of the acceleration measuring device 50 may transmit a walking start notification to the omnidirectional image capturing device 60 without going through the server 10.

  S8: The acceleration detector 52 of the acceleration measuring device 50 detects that the pedestrian has stopped walking based on the acceleration. For example, acceleration that is greater than or equal to the threshold is no longer detected in one or more of the x, y, and z directions, or the change in position in unit time obtained by integrating acceleration twice is less than the threshold, The end (stop) of walking can be determined by, for example. The determination may be made based on the angular velocity detected by the gyro sensor 207, the position detected by the GPS receiver 214, and the direction detected by the acceleration / direction sensor 206. By detecting the stop of walking, the acceleration detector 52 ends the measurement of acceleration.

  S9: The communication unit 51 of the acceleration measuring device 50 transmits a walk end notification to the server 10. Thereby, the server 10 can transmit an imaging end request to the omnidirectional image capturing device 60. Note that the communication unit 51 of the acceleration measuring device 50 may transmit a walking end notification to the omnidirectional image capturing device 60 without using the server 10. The subsequent processing may be the same as in FIG.

  According to the processing of FIG. 10, the measurement of walking motion can be started and ended without an instructor or the like performing an operation such as measurement start or measurement end.

<< Operation Variation >>
FIG. 11 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 11, differences from FIG. 8 will be described. In this embodiment, since it is effective to synchronize the time axes of measurement data and image data, in FIG. 8, the measurement start and imaging start, and the measurement end and imaging end are synchronized, respectively. Therefore, although absolute time is unnecessary, the server 10 may synchronize the measurement data and the image data using the absolute time as shown in FIG.

  S3-1: When the communication start part 51 of the acceleration measuring device 50 receives the measurement start request in step S2, the acceleration detecting part 52 of the acceleration measuring device 50 requests the server 10 for the start time.

  S3-2: Thereby, the acceleration detector 52 of the acceleration measuring device 50 can acquire the absolute time (measurement start time) at which the measurement is started from the server 10.

S5-1: When the communication unit 61 of the omnidirectional image imaging device 60 receives the imaging start request in step S4, the imaging unit 62 of the omnidirectional image imaging device 60 requests the server 10 for the start time.
S5-2: Thereby, the imaging unit 62 of the omnidirectional image imaging device 60 can acquire the absolute time (imaging start time) when imaging was started from the server 10.

S9-1: When the communication unit 51 of the acceleration measuring device 50 receives the measurement end request in step S9, the acceleration detecting unit 52 of the acceleration measuring device 50 requests the server 10 for the end time.
S9-2: Thereby, the acceleration detection unit 52 of the acceleration measuring device 50 can acquire the absolute time (measurement end time) when the measurement is ended from the server 10.

  S10-1: When the communication unit 61 of the omnidirectional image imaging device 60 receives the imaging end request in step S10, the imaging unit 62 of the omnidirectional image imaging device 60 requests the server 10 for the end time.

  S10-2: As a result, the imaging unit 62 of the omnidirectional image capturing apparatus 60 can acquire the absolute time (imaging end time) when the imaging is completed from the server 10.

  The server 10 acquires acceleration measurement start time and measurement end time, and pedestrian imaging start time and imaging end time together with measurement data and image data. Even if a time lag occurs between the start of acceleration measurement and the start of imaging for some reason, or a time lag occurs between the end of measurement of acceleration and the end of imaging, the server 10 can synchronize the time axes of the measurement data and the image data. it can.

<< Operation Variation >>
FIG. 12 is an example of a sequence diagram illustrating a modified example of the overall operation of the image display system 100. In FIG. 12, differences from FIG. 11 will be described. In FIG. 11, the server 10 transmits a measurement start request, an imaging start request, a measurement end request, and an imaging end request to the acceleration measuring device 50 and the omnidirectional image capturing device 60, respectively. In FIG. 12, the server 10 communicates with the acceleration measuring device 50 and transmits a measurement start request and a measurement end request.

  S2-1: Upon receiving the measurement start request, the communication unit 51 of the acceleration measuring device 50 transmits an imaging start request to the omnidirectional image capturing device 60.

  S2-2: The communication unit 61 of the omnidirectional image imaging device 60 receives the imaging start request, and the imaging unit 62 of the omnidirectional image imaging device 60 requests the acceleration measuring device 50 for the start time.

  S2-3: When the communication unit 51 of the acceleration measuring apparatus 50 is requested for the start time, the communication unit 51 requests the server 10 for the start time.

  S2-4: In response to this, the server 10 transmits the measurement start time to the acceleration measuring device 50.

  S2-5: The acceleration measuring device 50 transmits the imaging start time to the omnidirectional imaging device 60. Therefore, the acceleration measuring device 50 and the omnidirectional image capturing device 60 can treat the same start time as the measurement start time and the image capturing start time, respectively.

  Note that the time (current time) held by the acceleration measuring device 50 at the time of step S2-2 may be transmitted to the omnidirectional imaging device 60. The acceleration measuring device 50 starts measuring the acceleration from this time, and the omnidirectional imaging device 60 starts imaging from this time, so that the measurement start time and the imaging start time can be synchronized.

  The processing at the end of measurement and imaging is the same.

  S9-1: Upon receiving the measurement end request, the communication unit 51 of the acceleration measuring device 50 transmits an imaging end request to the omnidirectional image capturing device 60.

  S9-2: The communication unit 61 of the omnidirectional image imaging device 60 receives the imaging end request, and the imaging unit 62 of the omnidirectional image imaging device 60 requests the acceleration measuring device 50 for the end time.

S9-3: When the communication unit 51 of the acceleration measuring apparatus 50 is requested for the end time, the communication unit 51 requests the server 10 for the end time.
S9-4: In response to this, the server 10 transmits the measurement end time to the acceleration measuring device 50.
S9-5: The acceleration measuring device 50 transmits the imaging end time to the omnidirectional image capturing device 60. Therefore, the acceleration measuring device 50 and the omnidirectional imaging device 60 can treat the same end time as the measurement end time and the imaging end time, respectively.

  Note that the time (current time) held by the acceleration measuring device 50 at the time of step S9-2 may be transmitted to the omnidirectional imaging device 60. The acceleration measuring device 50 finishes measuring the acceleration at this time, and the omnidirectional imaging device 60 finishes the imaging at this time, so that the measurement end time and the imaging end time can be synchronized.

<Display example of image data and measurement data>
A display example of image data and measurement data will be described with reference to FIG. FIG. 13 shows an example of image data and measurement data displayed on the display 108 by the display device 30. A screen as shown in FIG. 13 is referred to as a data display screen 501. The data display screen 501 mainly has an image data field 502 and a measurement data field 503. The display control unit 32 displays the image data in the image data column 502 and displays the measurement data in the measurement data column 503. The display control unit 32 extracts the pedestrian imaging range from the image data of the omnidirectional image and displays it in the image data field 502. This process will be described later.

  The image data column 502 will be described. When the image data is a still image, when an instructor or the like presses the button 504, the operation input receiving unit 33 receives the operation, and the display control unit 32 displays the next image data in the image data column 502 in terms of time. Similarly, the display control unit 32 can display the previous image data in the image data column 502 by operating the button. In addition, the display control unit 32 may display image data as a still image in the image data column 502 one after another at a constant time interval.

  When the image data is a moving image, when an instructor or the like presses the button 504, the operation input receiving unit 33 receives the operation, and the display control unit 32 reproduces the image data as a moving image at the same reproduction speed as the imaging speed (number of frames / sec). To do. High-speed playback and low-speed playback can be stopped. When a leader or the like presses the button 504 again, the operation input receiving unit 33 receives the operation, and the display control unit 32 stops the reproduction. Therefore, a leader or the like can display image data at an arbitrary timing. Note that not only the image data but also sound collected together with the image data may be output.

  The measurement data column 503 will be described. In FIG. 13, accelerations in the x direction, y direction, and z direction are displayed in time series. An instructor or the like can display the acceleration in any one or more directions among the x direction, the y direction, and the z direction. Whether the image data is a still image or a moving image, the measurement data is continuously measured. Alternatively, in the case of a still image, the acceleration measuring device 50 and the omnidirectional image capturing device 60 are synchronized so that the acceleration measuring device 50 acquires acceleration at the timing when the omnidirectional image of the still image is captured. Good.

  The display control unit 32 scales and displays the measurement data in the measurement data column 503 so that all of the measurement data can be displayed in the measurement data column 503. Alternatively, measurement data for a certain amount of time may be displayed in the measurement data column 503 and switched at regular time intervals. When measurement data having a certain length is displayed in the measurement data column 503 as shown in FIG. 13A, the display control unit 32 performs measurement associated with the image data displayed in the image data column 502. A bar 505 indicating data is displayed in the measurement data column 503. For example, when the measurement data for the whole or a certain time is displayed in the measurement data column 503 and the image data of the moving image is reproduced, the display control unit 32 is associated with the frame of the image data being displayed. A bar 505 is superimposed on the measurement data. Therefore, the bar 505 moves the measurement data column 503 to the right with time or as the reproduction of the moving image proceeds. Conversely, when an instructor or the like moves the bar 505, the operation input accepting unit 33 accepts the operation, and the display control unit 32 displays an image data frame associated with the measurement data indicated by the moved bar 505. It is displayed in the data column 502.

  Moreover, you may display measurement data like FIG.13 (b). In FIG. 13B, only the measurement data associated with the image data displayed in the image data column 502 and the measurement data before and after that are displayed in the measurement data column 503. In the measurement data column 503, for example, only measurement data of about 1 second to several seconds is displayed at the same time. Note that the dotted measurement data indicates that the measurement data is continuous and is not displayed on the screen.

  In the case of FIG. 13B, the display control unit 32 displays the measurement data associated with the omnidirectional image displayed in the image data column 502 in the center of the measurement data column 503 as the moving image or still image progresses. To display. Since the image data and the measurement data are associated with each other by the server 10, at least the measurement data associated with the frame of the image data displayed in the image data column 502 can be displayed in the measurement data column 503. . When the moving image or still image in the image data field 502 is stopped, the measurement data is also stopped.

<< Evaluation of measurement data >>
When the measurement data is acceleration, peaks appear periodically and the peak height does not fluctuate so much in general walking. On the other hand, if the walking function declines due to age or injury, the peak of acceleration does not appear periodically or the height of the peak tends to fluctuate. The instructor or the like may be able to evaluate the walking motion only by looking at the image data, but the measurement data can support that the walking motion is not appropriate. In addition, an instructor or the like can specify a place where the measurement data is disturbed and can check the image data. Although the measurement data has a list property, since the image data is displayed only at a frame at a certain moment, an instructor or the like can search the image data for an inappropriate walking motion in a short time.

  In this embodiment, the acceleration is described as an example. However, whether or not the walking motion is appropriate depending on the angular velocity by the gyro sensor 207, the position detected by the GPS receiver, the direction detected by the acceleration / direction sensor, and the like. Can be supported. For example, the angular velocity measured by the gyro sensor 207 indicates the speed of change of the yaw angle, pitch angle, and roll angle when a pedestrian walks. In this case as well, it is known that, as with acceleration, peaks appear periodically and the peak height does not fluctuate much in general walking. Further, since the acceleration is obtained by differentiating the position twice, the position also gives information equivalent to the acceleration. In the case of turning walking, the azimuth constantly changes, but acceleration of the change in azimuth can be obtained by differentiating the azimuth twice. In this case as well, the peak period and height are a guide for evaluation. In addition, in these measurement data, if the peak is periodic and the peak fluctuation is extremely large even if the height fluctuation is small, the instructor may indicate that there is a possibility of improvement in walking motion. Can be confirmed.

  Further, the image rotation unit 34 of the display device 30 may calculate the area between the acceleration and the acceleration = 0 axis. For example, if you are walking in a circular shape, the y direction is the acceleration in the left-right direction of the pedestrian (the radial direction of the circle), so the area on the positive side and the area on the negative side are the acceleration in the radial direction relative to the line. Show shake. If a pedestrian walks on the line, the area on the positive side will be approximately equal to the area on the negative side. Therefore, it can be understood that the balance between the left leg and the right leg is poor when the area on the positive side and the area on the negative side are greatly different.

  Thus, a leader or the like can obtain a lot of information by viewing the measurement data and the image data whose time axes are synchronized.

<Rotation of spherical image>
Since the image data includes a 360-degree video, the entire image is distorted and the walking motion is difficult to understand. For this reason, it is preferable that the display control unit 32 displays only the range where the pedestrian is shown in the image data field 502. Such processing is expressed as rotation, trimming, extraction of pedestrians, or extraction of omnidirectional images. The rotation of the omnidirectional image refers to determining the range displayed in the image data field 502 by rotating the omnidirectional image horizontally (longitudinal direction).

  FIG. 14 is an example of a flowchart illustrating a procedure in which the image rotation unit 34 specifies a range in which a pedestrian is shown and displays the range in the image data field 502. The processing in FIG. 14 starts when, for example, the display device 30 receives image data and a moving image or a still image is played back.

  First, the image rotation unit 34 specifies the position of the pedestrian's foot (S10). Foot refers to the part of the footwear, from the heel to the toe when not wearing footwear. However, it may be detected as a foot below the waist or below the knee. In the case of a moving image, the image rotating unit 34 can detect a pedestrian by detecting a moving object if the pedestrian performs an operation such as stepping on several times. Or a pedestrian can be detected even if a face is recognized. There is a possibility that a person other than the pedestrian exists around the omnidirectional image capturing device 60, but the moving body occupying the largest pixel range is detected as a pedestrian. The pixel with the lowest latitude among the moving objects is the foot position. In addition, the image rotation unit 34 may learn a foot image by machine learning and recognize the foot from the omnidirectional image.

  Further, assuming that the pedestrian starts walking from a predetermined position, the initial position of the pedestrian is a predetermined position in the horizontal direction of the omnidirectional image. Therefore, in this case, the image rotation unit 34 can easily specify the pedestrian to be measured.

  Once the pedestrian can be identified from the omnidirectional image, the position of the pedestrian in the subsequent omnidirectional image should be able to be detected from the vicinity of the identified pedestrian, and can be easily detected in a relatively short time.

  In the case of a still image, the position of the pedestrian in the initial omnidirectional image is determined or input by a leader or the like. Or a pedestrian can be detected even if a face is recognized. A pedestrian can be detected from each still image by performing pattern matching on the subsequent omnidirectional image using the circumscribed rectangle of the pedestrian in the initial omnidirectional image.

  When the pedestrian starts walking, the image rotation unit 34 calculates the angular velocity of the pedestrian per unit time from the change in the foot position (S20). The angular velocity will be described with reference to FIG. FIG. 15 is a diagram schematically showing the position of the pedestrian's foot 511 in the top view of the omnidirectional image. For example, assume that the north direction is the reference. The foot 511 can take a position θ in the range of 360 degrees in the longitude direction (an angle with the north facing zero degree) θ. Since two legs 511 are detected in one frame, the average is set as the position of the legs 511. For example, the average change amount of the foot 511 in the past several seconds is calculated every predetermined time (for example, every time the frame is updated). Thereby, the angular velocity of a pedestrian with little fluctuation | variation can be calculated. The angular velocity can be referred to as walking speed (traveling speed information) viewed from the omnidirectional image capturing device 60. The walking speed in the x direction in real space is obtained by integrating the acceleration in the x direction.

  The image rotation unit 34 rotates the image at the angular velocity calculated in step S20 (S30). That is, when the frame is switched for the reproduction of the moving image, the image rotation unit 34 trims the image in the display range (predetermined range) of the image data column 502 around the position where the pedestrian moves at the angular velocity. Thereby, the pedestrian is reflected in the center of the image taken out by the image rotation unit 34.

  The display control unit 32 displays the image in the display range trimmed by the image rotation unit 34 in the image data field 502. Therefore, a leader or the like can evaluate the walking motion of the pedestrian without searching for the pedestrian from the omnidirectional image.

  In addition, since a camera with a normal angle of view is captured so that a pedestrian can be seen, processing such as rotation and trimming is not necessary. The server 10 or the omnidirectional image capturing device 60 may have the function of the image rotation unit 34. In this case, communication load and communication time when image data is transmitted from the server 10 or the omnidirectional image capturing device 60 to the display device 30 can be reduced.

<< Rotating spherical images by face recognition >>
In addition to paying attention to the position of the foot, the image rotation unit 34 may recognize a pedestrian by recognizing a face from the omnidirectional image and rotate the omnidirectional image.

  FIG. 16 is an example of a flowchart illustrating a procedure in which the image rotation unit 34 specifies a range where a pedestrian is shown and displays it in the image data field 502. The processing of FIG. 16 starts when, for example, the display device 30 receives image data and a moving image or a still image is reproduced.

  First, the image rotation unit 34 recognizes a face for each omnidirectional image (S10). FIG. 17 is an example of a diagram for schematically explaining the recognition of the face 512. The face 512 is recognized by setting an appropriate feature amount and learning by a learning identification device. For example, as feature quantities, Haar-like features, LBP (Local Binary Patterns) features, HOG (Histogram of Oriented Gradients) features, and the like are known. As learning methods of the learning identification device, SVM (Support Vector Machines), AdaBoost, and the like are known. However, the present invention is not limited to these, and it is sufficient that the face 512 can be recognized.

  The image rotation unit 34 trims the image in the display range 513 (predetermined range) in the image data field 502 around the face 512 from the omnidirectional image. Thereby, the pedestrian is reflected in the center of the image taken out by the image rotation unit 34.

  The display control unit 32 displays the image in the display range 513 trimmed by the image rotation unit 34 in the image data field 502. Therefore, a leader or the like can evaluate the walking motion of the pedestrian without searching for the pedestrian from the omnidirectional image.

  Instead of determining and trimming the position of the face 512 for each omnidirectional image, the image rotation unit 34 calculates the moving speed in the longitude direction from the positions of the faces 512 of several past omnidirectional images. Thus, the display range may be determined from the omnidirectional image to be displayed next. Since the position of the face 512 is less likely to fluctuate, a smooth moving image can be displayed.

  As described above, in the image display system 100 of the present embodiment, since a pedestrian turning around the omnidirectional image capturing device 60 or turning left and right is imaged, various walking motions of the pedestrian can be imaged. become. There is no need to move the camera or use motion capture. However, this is not to deny that a pedestrian wearing a marker is imaged. Further, since the acceleration of the pedestrian and the like are displayed on the same time axis as the image data, the instructor and the like can evaluate the walking motion using the image data and the measurement data in a complementary manner.

<Other configuration examples>
The configuration example in FIG. 2 is merely an example, and the image display system 100 may have the following configuration. FIG. 18A shows another example of a schematic configuration diagram of the image display system 100. In FIG. 18A, the server 10 has a display 108. In this configuration, the server 10 has the function of the display device 30 in FIG. 5, and the server 10 not only synchronizes the time axis but also rotates the omnidirectional image and displays measurement data and image data.

  In FIG. 18B, the server 10 and the display device 30 are eliminated. In this configuration, the acceleration measuring device 50 has the functions of the server 10 and the display device 30 of FIG.

  FIG. 19 is a sequence diagram in which the acceleration measuring device 50 displays measurement data and image data in the configuration as shown in FIG.

  S1: When starting measurement of walking motion, a leader or a pedestrian inputs an operation for starting measurement of walking motion to the acceleration measuring device 50. The acceleration measuring device 50 receives an operation of a leader or a pedestrian.

  S2: The acceleration measuring device 50 transmits an imaging start request to the omnidirectional imaging device 60.

  S3, S4: The acceleration measuring device 50 starts measuring acceleration, and the omnidirectional image capturing device 60 starts capturing a pedestrian.

  S5: When the walking is finished, the leader or the pedestrian inputs an operation to finish the measurement of the walking motion to the acceleration measuring device 50. The acceleration measuring device 50 receives an operation of a leader or a pedestrian.

  S6: The acceleration measuring device 50 transmits an imaging end request to the omnidirectional image capturing device 60.

  S7: The omnidirectional image capturing device 60 ends the image capturing and transmits the image data to the acceleration measuring device 50.

  S8: The acceleration measuring device 50 processes the measurement data as described in the server 10.

  S9: Further, the acceleration measuring device 50 synchronizes the time axes of the measurement data and the image data as performed by the server 10.

  S10: The acceleration measuring device 50 displays the measurement data and the image data on the display 215 of its own device.

  S11: Alternatively, the acceleration measuring device 50 may transmit the measurement data and the image data to a terminal such as a leader (such as the server 10).

  S12: Thereby, a terminal such as a leader displays measurement data and image data.

  Therefore, according to the configuration shown in FIG. 18B, the system configuration can be simplified. A pedestrian walks while wearing the acceleration measuring device 50. After the end of walking, the pedestrian can receive guidance from the instructor by passing the acceleration measuring device 50 to the instructor.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments. However, the present invention is not limited to these embodiments, and various modifications can be made without departing from the scope of the present invention. And substitutions can be added.

  For example, in the present embodiment, the measurement data and the image data are displayed after the measurement data is measured and the image data is captured. However, the measurement data and the image data may be displayed in real time. In this case, the displayed measurement data and image data are taken at almost the same time.

  Further, the configuration example such as FIG. 5 shown in the above embodiment is mainly for the purpose of facilitating understanding of the processing of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50. It is divided according to various functions. For example, a plurality of servers 10 may exist, and a plurality of servers 10 may cooperate to perform the processing of this embodiment. Further, the storage unit 1000 of the server 10 does not need to be included in the server 10, and the storage unit 1000 may be in a location where the server 10 can read and write data in the storage unit 1000.

  In FIG. 6, the functions of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 have been described by dividing them into several processing units. However, the present invention is not limited by the way of dividing or the name of each processing unit. The processing of the server 10, the display device 30, the omnidirectional image capturing device 60, and the acceleration measuring device 50 can be divided into more processing units depending on the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes.

  The omnidirectional image capturing device 60 is an example of a wide-angle image capturing device, the acceleration measuring device 50 is an example of a measuring device, the communication unit 11 is an example of an acquisition unit, and the image adjusting unit 12 is an example of an association unit. The display control unit 32 is an example of a display unit, the acceleration detection unit 52 is an example of a measurement unit, the image rotation unit 34 is an example of an extraction unit, and the synchronization instruction unit 15 is an example of an instruction unit. Yes, the operation input receiving units 33 and 14 are examples of receiving means.

DESCRIPTION OF SYMBOLS 10 Server 12 Image adjustment part 30 Display apparatus 32 Display control part 33 Operation input reception part 34 Image rotation part 50 Acceleration measuring apparatus 60 Omnispherical image imaging apparatus 100 Image display system

JP 2010-172394 A

Claims (13)

An image display system having an information processing device, a wide-angle imaging device capable of imaging a surrounding area in the horizontal direction, and a measuring device that measures information capable of evaluating a traveling operation of a traveling object,
Acquisition means for acquiring image data captured by the wide-angle imaging device and measurement data measured by the measurement device;
Association means for associating the image data with the measurement data;
An image display system comprising: display means for displaying the associated image data and the measurement data on a display device. The wide-angle imaging device captures the object traveling around the wide-angle imaging device while changing the direction or changing the direction,
2. The image display system according to claim 1, wherein the association unit associates the measurement data with the image data obtained by imaging the target moving while changing the direction or changing the direction. 3. The information processing apparatus has instruction means for requesting the measurement apparatus to start measurement, requesting the wide-angle imaging apparatus to start imaging, requesting the measurement apparatus to end measurement, and requesting the wide-angle imaging apparatus to end imaging. ,
The associating means associates the measurement data with the image data whose imaging data and the measurement data measurement start are substantially the same, and the image data imaging completion and the measurement data measurement completion are substantially the same. The image display system according to claim 1 or 2. The display device includes a receiving unit that receives at least one of the start of measurement of the measurement data or the start of imaging of the image data,
The image display system according to claim 3, wherein at least one of the start of measurement of the measurement data received by the reception unit and the start of imaging of the image data is notified to the information processing apparatus. The measurement device starts measurement of the measurement data when detecting the measurement data having a size greater than or equal to a threshold, and notifies the wide-angle imaging device that the measurement data having a size greater than or equal to the threshold has been detected. When the wide-angle imaging device is notified that the measurement data having a size equal to or larger than a threshold is detected, the imaging of the image data is started.
The measurement device ends the measurement of the measurement data when the size of the measurement data is less than a threshold, and notifies the wide-angle imaging device that the size of the measurement data is less than the threshold, The wide-angle imaging device ends imaging of the image data when notified that the size of the measurement data is less than a threshold,
The associating means associates the measurement data with the image data whose imaging data and the measurement data measurement start are substantially the same, and the image data imaging completion and the measurement data measurement completion are substantially the same. The image display system according to claim 1 or 2. The measurement device acquires a measurement start time and a measurement end time of the measurement data,
The wide-angle imaging device acquires an imaging start time and an imaging end time of the image data;
The image display system according to claim 1, wherein the acquisition unit acquires the measurement data together with the measurement start time and the measurement end time, and acquires the image data together with the imaging start time and the imaging end time. By detecting the target of the image data, the traveling speed information when the target travels around the wide-angle imaging device is calculated, and an image in a predetermined range determined by the traveling speed information is calculated from the image data. Having extraction means for extracting;
The image display system according to claim 1, wherein the display unit displays the image in the predetermined range extracted by the extracting unit on the display device. An extraction means for detecting the target of the image data and extracting an image of a predetermined range including the target from the image data;
The image display system according to claim 1, wherein the display unit displays the image in the predetermined range extracted by the extracting unit on the display device.   The said wide-angle imaging device alert | reports that the imaging of the said image data is started, or the said measuring device alert | reports starting the measurement of the said measurement data. Image display system. An information processing device capable of communicating with a wide-angle imaging device capable of imaging the surroundings in the horizontal direction and a measuring device that measures information capable of evaluating a traveling operation of a traveling object,
Acquisition means for acquiring image data captured by the wide-angle imaging device and measurement data measured by the measurement device;
Association means for associating the image data with the measurement data;
An information processing apparatus comprising: display means for displaying the associated image data and the measurement data on a display device. An information processing apparatus capable of communicating with a wide-angle imaging device capable of imaging a horizontal periphery,
A measuring means for measuring information that can be used to evaluate the progress of the target object,
Acquisition means for acquiring image data captured by the wide-angle imaging device;
Association means for associating the image data with the measurement data measured by the measurement means;
An information processing apparatus comprising: display means for displaying the associated image data and the measurement data on a display device. An information processing device capable of communicating with a wide-angle imaging device capable of imaging the surroundings in the horizontal direction and a measuring device that measures information capable of evaluating a traveling operation of a traveling object,
Acquisition means for acquiring image data captured by the wide-angle imaging device and measurement data measured by the measurement device;
Association means for associating the image data with the measurement data;
A program for causing a display device to display the associated image data and the measurement data on a display device. An information processing device that can communicate with a wide-angle imaging device capable of imaging the surroundings in the horizontal direction,
A measuring means for measuring information that can be used to evaluate the progress of the target object,
Acquisition means for acquiring image data captured by the wide-angle imaging device;
Association means for associating the image data with the measurement data measured by the measurement means;
A program for causing a display device to display the associated image data and the measurement data on a display device.

Images