JPWO2020158955A1 - Information processing equipment - Google Patents

Abstract

Provided is an information processing device capable of taking a picture in a free posture while visually recognizing a camera viewpoint image. The imaging means performs imaging. The text data generation means generates text data corresponding to the character string based on the captured image captured by the imaging means. The presenting means presents both the captured image captured by the imaging means and the text data.

Description

The present invention relates to an information processing device.

Conventionally, in mobile terminals with cameras such as smartphones, in addition to the rear camera (camera dedicated to taking pictures of the other party) facing the back side of the terminal display, the front camera (camera dedicated to taking selfies) facing the display side. Those equipped separately were widespread.
In such a terminal, if two high-performance cameras are used, the terminal will be expensive, the camera and lens must be large, which is not preferable in terms of design, and usually the other party is taken. Since it is used as the main camera, a small, inexpensive, low-performance front camera was used as the sub camera.

However, there is also a need to take a selfie with a high-performance camera while visually recognizing the image actually taken by the camera (hereinafter referred to as "camera viewpoint image") on the display unit, so it is called the other party shooting mode. In the selfie mode, there was one that shared the camera.
For example, Patent Document 1 discloses a technique in which an image sensor unit is rotated by a movable mechanism according to a selection of a shooting mode, and a filter and a lens used are automatically switched according to the rotation.

Japanese Unexamined Patent Publication No. 2008-131555

However, in the conventional technology including Patent Document 1, since the imaging direction is extremely limited, functionally, there is no difference from the case where two or three high-performance cameras are attached with the eyes closed to the problem of cost. There was a problem that the posture of the user at the time of shooting was limited.

The present invention has been made in view of such a situation, and it is possible to point a high-performance camera in any direction with respect to the main body of the device, and to take a picture in a free posture while visually recognizing a camera viewpoint image. The purpose is to provide a portable information device capable of providing a portable information device.

In order to achieve the above object, the information processing apparatus according to one aspect of the present invention is
An imaging means for imaging and
A subject extraction means for extracting a subject from an captured image generated by the image pickup means, and a subject extraction means.
An information extraction means for extracting information indicated by the subject, and
Information analysis means for analyzing the information and
A generation means for generating an object based on the result of the analysis,
A presentation means for presenting the object together with the captured image,
To prepare for.

According to the present invention, it is possible to provide a portable information device capable of taking a picture in a free posture while visually recognizing a camera viewpoint image.

It is a figure which shows the appearance structure of one Embodiment of the information processing apparatus of this invention. It is a figure which shows the appearance structure of the information processing apparatus of FIG. It is a figure which shows the state which the connecting part of the information processing apparatus of FIG. 1 was rotated. It is a figure which shows the state which the stand part of the information processing apparatus of FIG. 1 was rotated. It is a figure which shows the state of using the information processing apparatus of FIG. 1 leaning against a tabletop. It is a figure which shows an example of the photographed image selection screen to be displayed on the information processing apparatus of FIG. It is a figure which shows the state that the information processing apparatus of FIG. 1 is performing indoor navigation of a store or the like. It is a figure which shows the state that the information processing apparatus of FIG. 1 is issuing an advertisement in consideration of a position and a direction of a camera. It is a figure which shows the state which the street view is updated in real time by the information processing apparatus of FIG. It is a figure which shows the state which the information processing apparatus of FIG. 1 is in a horizontal position, and the information of the top and bottom (top and bottom) at the time of image pickup by a camera which is directed to the direction different from the main body is added to a photograph, and how to display. .. It is a figure which shows the state which the information processing apparatus of FIG. 1 is in a vertical position, and the information of the top and bottom (top and bottom) at the time of image pickup by a camera which is directed to the direction different from the main body is added to a photograph, and how to display. .. It is a figure which shows the state which the shutter remote controller is removed from the main body in the information processing apparatus of FIG. It is a figure which shows the state that the mirror is set in front of the front camera of the information processing apparatus equipped with a fixed camera, which is different from the information processing apparatus of FIG. 1, and the book is automatically translated and the receipt data is read. .. It is a figure which shows the state of playing an AR game while walking outdoors with the information processing apparatus of FIG. It is a figure which shows the state which the sound source is tracked and photographed by the information processing apparatus of FIG. It is a figure which shows the state which the information processing apparatus of FIG. 1 uses a problem described in a paper medium or the like for education. It is a figure which shows the state in which the information processing apparatus of FIG. 1 is self-sustainingly stabilized by a foldable stand. It is a figure which shows the state which the information processing apparatus of FIG. 1 is searching for the saved photograph based on the position information. It is a figure which shows the state that the information processing apparatus of FIG. 1 displays information and is searching for the corresponding photograph from the WEB based on the place. It is a figure which shows each state which changed the shape in the case where the information processing apparatus includes a slide type rotary camera. It is a figure which shows the structure of the biaxial rotation mechanism part of the information processing apparatus of FIG. It is a figure which shows each state which changed the shape in the case where the slide type rotary camera is provided in the notch part of the display part of the information processing apparatus of FIG. It is a figure which shows an example of the translation screen (UI) of the information processing apparatus of FIG. It is a flowchart which shows the process of translation and reading aloud of the information processing apparatus of FIG. It is a flowchart which shows the process of translation and reading aloud of the information processing apparatus of FIG. It is a figure which shows an example of the reading aloud screen (UI) of the information processing apparatus of FIG. It is a flowchart which shows the process of reading aloud of the information processing apparatus of FIG. It is a flowchart which shows the process of reading aloud of the information processing apparatus of FIG. It is a flowchart which shows the process of textification of the information processing apparatus of FIG. It is a flowchart which shows the process of textification of the information processing apparatus of FIG. It is a figure which shows an example of the mirror unit attached to the information processing apparatus of FIG. It is a figure which shows another example of the mirror unit attached to the information processing apparatus of FIG.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing an external configuration of an embodiment of the information processing apparatus of the present invention. FIG. 1A is a front view of the information processing apparatus. FIG. 1B is a top view of the information processing apparatus 1. FIG. 1C is a left side view of the information processing apparatus.
The information processing apparatus 1 has at least a display function and a digital camera function, and is composed of a main body portion 11 and a connecting portion 12, which are rotatably connected to each other.

The main body portion 11 is a first housing that is formed in a short columnar shape of a planar quadrangle and contains various substrates and the like. A display unit 13 composed of a touch panel type display or the like is arranged on a predetermined one surface 11a (hereinafter referred to as "front surface 11a") of the main body unit 11.

FIG. 2 is a diagram showing an external configuration of the information processing apparatus of FIG. 1. FIG. 2A is a rear view of the information processing apparatus. FIG. 2B is a cross-sectional view taken along the line AA of FIG. 2A. FIG. 2C is a cross-sectional view of FIG. 2A as viewed from above.

The back surface 11b of the main body 11 is one surface on the side facing the front surface 11a described above in FIG.
The connecting portion 12 is formed in a substantially rectangular planar shape, and is rotatably arranged on one short side side of the back surface 11b of the main body portion 11. The connecting portion 12 has a camera base portion 12-1 and a camera-oriented portion 12-2. A circular camera 21 is arranged in front of the camera-oriented portion 12-2 (the surface on the side visible in FIG. 2A). That is, the connecting portion 12 is a second housing in which the camera 21 is built. The camera 21 is an imaging unit that captures an image of a subject. The captured image including the subject captured by the camera 21 is displayed on the display unit 13. Here, as shown in FIG. 2C, the recess width W is the width of the recess for the camera unit. The turning radius r2 is the longest turning radius from the turning axis (center) of the second hinge. As a result, the relationship between the recess width W and the turning radius r2 becomes the relationship shown by the following equation (1).
W / 2> r2 ... (1)

The camera base portion 12-1 is rotatably supported by the entire connecting portion 12 with respect to the main body portion 11 with the rotation shaft Z1 as the rotation center.
More specifically, the camera base 12-1 is configured to be able to rotate about the rotation axis Z1 while sliding the end surface of the pillar-shaped outer shape to the main body 11.

It is preferable that the shaft support penetrates between both end faces in terms of strength. ..
The camera-oriented unit 12-2 on which the camera 21 is mounted is rotatably cantilevered with respect to the camera base 12-1 with the rotation axis Z3 arranged in a direction substantially orthogonal to the rotation axis Z1 as the rotation center. Axial support.

Here, the arrangement direction of the camera 21 is approximately 90 degrees with respect to the rotation axis Z3.
That is, the camera 21 can take a motion on the solid angle with respect to the main body 11 by two substantially orthogonal and independent rotation axes, the rotation axis Z1 and the rotation axis Z3.

In this way, the rotation axis Z1 that changes the angle formed by the optical axis direction of the camera 21 and the normal direction of the display unit 13 and the rotation in a direction different from the rotation axis Z1 (in this example, substantially orthogonal directions). The connecting portion 12 is a portion that directly or indirectly connects the camera 21 with respect to the main body portion 11 by a plurality of rotating shafts including the shaft Z3.
The connecting portion 12 has one or more movable portions having a volume smaller than that of the main body portion 11. In this example, a camera base portion 12-1 and a camera-oriented portion 12-2 are provided as such movable portions.
It suffices if the camera 21 as an image pickup unit is arranged on a predetermined surface of one or more movable parts, but in this example, it is arranged in front of the camera-oriented unit 12-2 as described above.
The angle between the direction of the rotation axis Z1 and the direction of the rotation axis Z3 is approximately 90 degrees.

The information processing apparatus 1 is further provided with a stand 14 so that the information processing apparatus 1 can be placed leaning against a desk or the like.
The stand 14 is rotatably supported with respect to the main body 11 with the rotation shaft Z2 as the rotation center.
That is, the stand 14 is formed in a frame shape and forms a suitable arbitrary angle with the main body 11 by rotation about the rotation axis Z2 to independently stabilize the information processing apparatus 1 on a horizontal plane (described later). See Figure 5).
It is needless to say that the illustrated planar U-shape is an example and may be a planar U-shape or the like.
Further, if it can be used on a vertical surface described later, it may have a planar hook shape or the like.

By forming the stand 14 so as to surround the connecting portion 12 in a frame shape, the space efficiency can be significantly improved as compared with arranging the stand 14 and the connecting portion 12 at separate positions, which is a wasteful information processing device. The housing of 1 can be prevented from becoming large. Alternatively, the internal board can be prevented from being unnecessarily divided.
The connecting portion 12 and the stand 14 substantially share the center line of both end faces with respect to the rotation axis like the lines AA.

Further, the rotation shafts Z1 to Z3 will be described below.
In the rotation shaft Z1, the main body portion 11 and the connecting portion 12 are the rotation centers of the connecting shaft, and are arranged in parallel with respect to the lateral direction of the display unit 13.
The rotation range of the rotation shaft Z1 is approximately 180 degrees or more from the back surface 11b, and it is desirable that the rotation range exceeds 270 degrees.

The rotation shaft Z2 is the rotation center of the connection shaft between the main body 11 and the stand 14.
It is desirable that the rotation range of the rotation shaft Z2 reaches 180 degrees from the back surface 11b, and in particular, any angle from the back surface 11b to 90 degrees is opposed to the self-weight moment of the information processing device 1 on the horizontal plane. An appropriate rotation resistance may be given so that it can be maintained.
As a result, the user can arrange the information processing apparatus 1 independently and stably at a desired angle posture on a horizontal plane.
It should be noted that a portion to be offset may be provided on the rotation shaft Z2 side of the stand 14 so that the rotation center of the rotation shaft Z1 and the rotation shaft Z2 can be shared.

The rotation shaft Z3 is the rotation center of the connection shaft between the camera base portion 12-1 and the camera orientation portion 12-2, and is arranged in parallel with the longitudinal direction of the display unit 13.
The rotation shaft Z3 is provided substantially orthogonal to the rotation shaft Z1 and rotates on a substantially orthogonal plane around the rotation shaft Z1.
It is desirable that the rotation range of the rotation shaft Z3 reaches 180 degrees to the left and right from a substantially orthogonal plane around the rotation shaft Z1 in which the rotation shaft Z3 rotates.

FIG. 3 is a diagram showing a state in which the connecting portion of the information processing apparatus of FIG. 1 is rotated.
In FIG. 3A, the connecting portion 12 (more accurately, the camera base portion 12-1) of the information processing apparatus 1 of FIG. 1 is in a state of being rotated 90 degrees about the rotation axis Z1.
In FIG. 3B, the connecting portion 12 (more accurately, the camera base portion 12-1) further rotates about the rotation axis Z1 from the state of FIG. 3A, and is centered on the rotation axis Z1. It is in a state of being rotated 180 degrees.
In FIG. 3C, the connecting portion 12 (more accurately, the camera-oriented portion 12-2) is in a state of being further rotated 90 degrees about the rotation axis Z3 from the state of FIG. 3B. ..
As shown in FIGS. 3A to 3C, the distance h1 is information processing from the rotation axis (center) of the first axis hinge to the upper side surface of the main body 11 of the information processing device 1 (smartphone). This is the longest distance to the top surface of the device 1 (the one with the display unit 13) or the bottom surface of the information processing device 1 (the one without the display unit 13). Further, the distance h2 is the distance from the rotation axis (center) of the hinge of the first axis to the rotation cross section of the hinge of the second axis. The relationship between the distance h1 and the distance h2 is expressed by the following equation (2).
h2> h1 ... (2)

FIG. 4 is a diagram showing a state in which the stand 14 of the information processing apparatus 1 of FIG. 1 is rotated.
FIG. 4A is in the same state as in FIG. 3B described above, and the stand 14 is not rotated and is housed in the main body 11.
In FIG. 4B, the stand 14 is further rotated about the rotation shaft Z2 from the state of FIG. 4A, and is in a state of being rotated 180 degrees around the rotation shaft Z2.
Further, as in FIG. 3, the relationship between the distance h1 and the distance h2 is the same as that in the above equation (2).

By rotating the stand 14 180 degrees around the rotation axis Z2 as shown in FIG. 4 (B), the user can clip the stand 14 to the wall KB as shown in FIG. 4 (C). The information processing device 1 can be attached to the wall KB by applying it to CL or the like.
In the state of FIG. 4C, the optical axis direction of the camera 21 (shooting direction of the subject) and the normal direction of the display unit 13 (direction in which the user looks at the display unit 13) are normal directions with respect to the surface of the wall KB. It has become.
Therefore, by being located facing the wall KB, the user can easily take a picture of himself / herself, that is, a selfie shot (selfie) while looking at the camera viewpoint image (the image of himself / herself). ..

As described above, the stand 14 is rotated with respect to the main body 11 by the rotating shaft Z2, and the camera 21 (connecting portion 12) is displayed by the display unit 13 (main body) by the two axes of the rotating shaft Z1 and the rotating shaft Z3. It rotates with respect to the part 11).
Therefore, the user can use the information processing device 1 by hanging it on the wall KB as shown in FIG. 4 when taking a selfie shot (selfie), or the information processing device 1 can be used as shown in FIG. It can also be used by leaning against the table.

FIG. 5 is a diagram showing a state in which the information processing apparatus 1 of FIG. 1 is leaned against a tabletop and used.
FIG. 5A is a perspective view from a viewpoint looking at the front surface 11a side on which the display unit 13 is arranged.
FIG. 5B is a perspective view showing a desktop usage pattern of the information processing apparatus of FIG. 1, which looks at the back surface 11b side on which the display unit 13 is arranged. The stand 14 in this embodiment is in a state of being rotated within 90 degrees around the rotation axis Z2 from the state of FIG. 4A.

Even if the rotation shaft Z2 is provided with an appropriate rotation resistance so that an arbitrary angle from the back surface 11b to 90 degrees can be maintained against the own weight moment of the information processing apparatus 1 on the horizontal plane. good.
As a result, the user can arrange the information processing apparatus 1 independently and stably at a desired angle posture on a horizontal plane.

As shown in FIG. 5, the user related to the information processing apparatus 1 of FIG. 1 is in a state where the main body 11 and the stand 14 are deformed so as to expand by a predetermined angle when taking an image of himself / herself. 1 is erected on a desk or the like.
Even in this state, the information processing apparatus 1 can point the camera 21 at the subject by rotating the rotation shaft Z1 and the rotation shaft Z3.

Further, by using the shutter remote control function and the self-timer function together, the information processing device 1 can take a picture without holding the main body 11 by hand, so that various problems described later can be solved.

By adopting the information processing apparatus 1 to which the present invention described above is applied, it is possible to obtain various effects as described below.

That is, selfie shots (selfie shots) using the front camera of conventional smartphones are popular all over the world, especially in the Asian region.
However, conventional smartphones have various problems as described below.
By adopting the information processing apparatus 1 to which the present invention is applied, it is possible to achieve the effect of solving these various problems.

For example, users of conventional smartphones press the shutter with their own hands, so they cannot take pictures of themselves from a distance longer than the length of their own hands. In other words, because of the wide-angle limit, it may not be possible for everyone (family, lover, friends, etc.) to take a picture together, or it is impossible to take a full-body picture of yourself.
As a conventional solution to this problem, there is a method of using a selfie stick. With this method, you can take pictures from a distance longer than the length of your hand, but you cannot take pictures from a distance longer than the length of the selfie stick. In addition, there are places where it is necessary to carry around the self-stick, and the use of the self-stick is prohibited from the viewpoint of safety (theme parks, sports facilities, concert venues, station platforms, etc.), which is a versatile solution. is not it.
In other words, there is a problem of wanting to shoot without using a selfish stick. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing apparatus 1 to which the present invention is applied has the stand 14, it can be fixed to a wall or a table as shown in FIGS. 4 and 5, and no selfish stick is required.

Also, since I reach out my hand, I can only shoot with the hand that pressed the shutter extended, resulting in an unnatural pose.
This may be alleviated by using a selfish stick, but the unnaturalness may remain and the pose that the white part wants to take (for example, a pose using both hands) may not be possible.

In other words, there is a problem of wanting to shoot without using a selfish stick. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing device 1 to which the present invention is applied has a stand 14, a self-timer, and a shutter remote control function, it can be fixed to a wall or a table as shown in FIGS. 4 and 5, and is selfish. Is no longer needed.

Also, when it comes to a larger number of SELPHY, it becomes a group photo rather than SELPHY, but there are cases where not everyone can fit in the camera, whether it is their own hands or SELPHY Stick.
In such a case, one of the companions will be sacrificed and become the photographer, but the photographer cannot enter the group photo.
When holding a party at a store, it is often the case to ask a clerk to take a group photo, but it is troublesome because it is necessary to call the clerk.

That is, there is a problem that all the people to be photographed should be included in the photograph without bothering the hands of people other than the person to be photographed. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing device 1 to which the present invention is applied has the stand 14 and the biaxial rotation mechanism of the camera, it can be fixed to a wall or a table as shown in FIGS. 4 and 5, and can be photographed. The hands of people other than the target person are no longer required.

Also, for example, when checking the form of sports by yourself, I want to shoot with the camera of the smartphone, but if there is nothing to fix the smartphone on the floor or desk, I can not even shoot, so I shoot with the smartphone I often give up.

Similarly, the smartphone may be placed on a desk or the like to take a selfie shot, but when the front camera is pointed at the subject and the smartphone is placed, it cannot be placed stably or it takes time to install.

In other words, there is the problem of wanting to shoot without using a tripod. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing device 1 to which the present invention is applied has the stand 14 and the two-axis rotation mechanism of the camera, it can be fixed to a wall or a table as shown in FIGS. 4 and 5, and is a tripod. Is no longer needed.
In addition, if you want to shoot on a vertically long screen, you can shoot while the stand 14 is upright and the connecting portion 12 is rotated 180 degrees or more so that it is fixed on the table, eliminating the need for a tripod. In this case, it is not always necessary to stand the stand 14.

Also, when video chatting on a smartphone, many people need to have the smartphone with them throughout the chat in order to see their face on the front camera of the smartphone.

According to the survey, 80% of people say that they are chatting with their hands in their hands, and depending on the time they chat, their hands get tired.

In other words, there is the problem of wanting to have a video chat without holding a smartphone. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing device 1 to which the present invention is applied has the stand 14 and the rotation mechanism of the camera, it can be fixed to a wall or a table as shown in FIGS. 4 and 5, and the smartphone can be held by hand. It becomes unnecessary to have it.
In addition, if you want to take a picture of your face on a vertically long screen and have a video chat, you can take a picture while fixing it on the table by standing the stand 14 and rotating the connecting part 12 by 180 degrees or more, so you can take a picture with a smartphone. It is no longer necessary to hold it in your hand. In this case, it is not always necessary to stand the stand 14.

In addition, as a solution to these problems, there is a method of mounting a stand on the smartphone to place the smartphone on a desk, but the angle between the display screen and the front camera is fixed by the stand, so the subject is not always the subject. Again, this is not a versatile solution for the purpose of taking a picture or video with the camera not pointing in the direction.

That is, there is a problem that it is desired to shoot without using a front camera with a fixed angle. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing device 1 to which the present invention is applied has the stand 14 and the two-axis rotation mechanism of the camera, the rear camera is rotated to the front by two-axis rotation as shown in FIGS. 4 and 5. It can be turned around and pointed at any angle, eliminating the need for a front camera with a fixed angle.
In addition, if you want to shoot on a vertically long screen, you can shoot while the stand 14 is upright and the connecting part 12 is rotated 180 degrees or more so that it is fixed on the table, eliminating the need for a front camera with a fixed angle. Become. In this case, it is not always necessary to stand the stand 14.

Even if the smartphone is equipped with a stand, it cannot be placed if there is no place to put it and there is only a wall.
As a solution to this problem, there is a method of hanging an information processing device on the wall and fixing it, but it is necessary to prepare a hook for that purpose on the smartphone, which also requires space in the smartphone body.

That is, there is a problem that the hook is not provided separately. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, the information processing apparatus 1 to which the present invention is applied utilizes the gap provided between the stand 14 and the connecting portion 12 by rotating the connecting portion 12 and the stand 14 by 180 degrees. Since it can be suspended and supported on a flat surface by using a protrusion or a pin-shaped member, it is suitable for use on a wall, and it is not necessary to separately provide a hook.

Also, the front camera is generally compared to the rear camera,
(A) Low resolution (B) Low sensitivity (Dark scenes cannot be shot without a flash)
(C) Fixed focus, not autofocus (d) Autofocus speed is slow (e) Low zoom magnification (f) Without a flashlight, many are inferior in performance. , I can't take pictures with beautiful image quality.
In addition, if the performance of the rear camera and the front camera are different, it is necessary to develop two cameras for one model.
As a solution to this problem, there is application software that takes self-shots with the rear camera, but since the display is not visible, it is very inconvenient to navigate by voice.

In other words, there is a problem that application software for taking self-shots without looking at the display unit does not want to be used. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, in the information processing device 1 to which the present invention is applied, the camera 21 as a high-performance camera can be directed in any direction with respect to the device body, and the camera can be viewed in a free posture while visually recognizing the camera viewpoint image. Since it is possible to take a picture with the camera, application software for taking a selfie shot without looking at the display unit 13 becomes unnecessary.

In addition, the stand and the rotary camera, which are the above solutions, also require each axis for rotation on the main body side, and it is necessary to secure space for the two axes, and the housing of the main body. Will be bigger. In addition, although it depends on how the camera is rotated, even if the camera of the smartphone that stands up on the stand rotates, the camera often does not face the subject, which is often not a solution. It is possible to increase the axis of rotation so that the camera faces the subject, but it is often troublesome to always point the camera at the subject when taking a picture.

That is, there is a problem that the camera is always aimed at the subject with a small number of rotation axes. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing apparatus 1 to which the present invention is applied may use the rotation axes of the stand 14 and the camera 21 in combination, as shown in FIGS. 4 and 5, the camera is always rotated by only two axes. It is possible to point 21 and it is not necessary to increase the number of rotation axes.

In addition, with the conventional information processing device in which both the rear camera and the front display are fixed to the main body, high-angle shots taken from a position higher than the eyes and low-angle shots taken from a position lower than the waist are shot by the camera. Since the orientation and the orientation of the display unit are opposite to each other, it is difficult to take a picture while looking at the display unit.
For example, a high-angle shot is necessary when taking a picture, when the subject is stuck in a crowd, or when shooting from above the crowd, but when the camera is pointed at the subject, the display section is on top. It turns to face and it is not possible to shoot while visually recognizing the camera viewpoint image.

Similarly, if the camera is pointed at a subject located higher than the camera from a position lower than the waist, the display unit faces downward, and it is still impossible to take a picture while visually recognizing the camera viewpoint image.
As a solution to this problem, when the camera is rotated, the orientation of the camera and the orientation of the display unit can be set separately, so that it is possible to take a picture while visually recognizing the camera viewpoint image.

At this time, if you rotate the camera, the vertical direction of the screen may change, but if you do not devise anything, the vertical direction may be reversed.
In such high-angle shots and low-angle shots, if the direction of rotation of the camera is fixed (such as rotating only clockwise from the normal position), it is difficult to shoot, so it can rotate in both directions. Is desirable.

That is, there is a problem that it is desired to shoot without using a rear camera with a fixed angle. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing device 1 to which the present invention is applied has a biaxial rotation mechanism of the stand 14 and the camera 21, the rear camera 21 is biaxially rotated as shown in FIGS. 4 and 5. This makes it possible to direct the camera up and down at any angle, eliminating the need for a rear camera with a fixed angle.
When the camera 21 is rotated, it is desirable that the angle of view (wide-angle, normal, narrow-angle, etc.) that can be photographed can be automatically switched at some timing. Generally, the distance to the subject is short on the optical axis side of the front camera, so I want to shoot at a wide angle, but I want to shoot at a normal or narrow angle on the optical axis side of the rear camera. Therefore, it is desirable that the angle of view changes automatically depending on the orientation of the camera.

Also, when taking a selfie shot, it is necessary to press the shutter button or touch panel to release the shutter, and it is not possible to hold the smartphone firmly, and the unstable smartphone may be dropped.
As a solution to this problem, there is a timer setting and shutter pressing by gesture recognition or voice recognition, but it is difficult to understand when the shutter is released or the reaction is bad, so for example, take a picture of the moment when five people jump. It is difficult to use when you want to.

In other words, there is a problem that you want to take a picture by releasing your hand from the smartphone and pressing the shutter button. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, since the information processing device 1 to which the present invention is applied has a shutter remote control compatible function, as shown in FIGS. 4 and 5, when taking a selfie shot, the shutter is not pressed without pressing the shutter button or the touch panel. It is possible to lower the smartphone, eliminating the need to hold the smartphone firmly.

Here, even if there is a shutter remote control compatible function, for example, when a group shot is taken by 10 people, the above-mentioned information processing device 1 is installed on a desk or the like, and the camera 21 is rotated toward 10 people. If the information processing device 1 is far away, the display unit 13 becomes too small to be seen, and it may be difficult to release the shutter because it may not be possible to see if everyone fits in the frame of the camera viewpoint image or the facial expressions of each may not be known. be.
That is, there is a problem that one wants to take a picture while checking the camera viewpoint image by some method.
The solution to this problem is to adopt the following method. That is, for the information processing device 1 to which the present invention is applied, a screen sharing function that transfers a camera viewpoint image to another information processing device such as a smartphone or a personal computer by wire or wirelessly and enables operation is provided. It is advisable to adopt the method of mounting.
This makes it possible to shoot a still image or a moving image from a distant position while sharing the camera viewpoint image of the information processing device 1 with another information processing device. In some cases, the information processing devices are directly connected to each other, or the camera viewpoint image of the information processing device 1 placed at home is shared with the personal computer in the office by using the Internet line, and the screen of the personal computer is viewed at home. It is also possible to take pictures of the remaining pets.

Also, with a conventional digital camera or smartphone, you can take a composite panoramic shot by moving the camera sideways.
However, it is necessary to shoot while moving the smartphone or camera sideways in various directions, and it is often not possible to combine them neatly.

That is, there is a problem that the camera wants to take a picture of the surroundings without moving the main body sideways. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, in the information processing apparatus 1 to which the present invention is applied, the rotation of the rotation axes Z1 and Z3 of the camera 21 may be electric, so that the camera 21 is panned while the main body is stopped as shown in FIG. It is possible to make the camera 21 move laterally, and it is not necessary to move the camera 21 laterally.

In addition, there have been attempts to utilize human gestures in the operation of devices and games, but since the camera that recognizes gestures must always face the subject, a camera device and a game device are prepared separately. I had to do it, and I couldn't handle it with one smartphone.
Moreover, the camera for gesture recognition is very expensive, and an additional cost is incurred.

In other words, there is a problem of wanting to recognize gestures without using a camera dedicated to gesture recognition. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, the information processing apparatus 1 to which the present invention is applied is possible if the camera 21 aimed at a person has high functionality (for example, gesture recognition of exhibition), and a camera dedicated to gesture recognition becomes unnecessary.

In addition, smartphones have traditionally had navigation software for pedestrians, but even so, especially outdoors, it is not easy to know the direction you are facing, and you do not know or know the direction you should go. It takes time.
As a solution to this problem, there is AR navigation using a camera.

The navigation can be used immediately without taking a long time to recognize the direction by the user interface that superimposes the arrow on the image actually captured by the camera.
However, in order to shoot with a camera fixed at an angle orthogonal to the back of the main unit, it is necessary to always hold the smartphone main unit perpendicular to the ground, and the display unit at that time may be at a very difficult angle to see. I'm tired of having it all the time.

"Walking smartphone", which is the act of using a smartphone while walking outdoors, is also a social phenomenon, but since it is concentrated on the screen, it is hard to notice until it is about to collide.
These two problems may be solved if there is a rotary camera.
During "Walking smartphone", the image of the rotated camera 21 can be displayed on the display unit 13 as one of the windows, and the existence of obstacles can be communicated by sound, screen or vibration, and danger can be avoided. ..

In this embodiment, since the camera rotates in the vertical direction with respect to the display surface, the above two solutions can be obtained, but if the direction of the camera rotates smoothly, the camera is idle and unstable. There is also.

That is, even if the main body is gripped and stabilized, there is a problem that the camera is idle and unstable. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, the information processing apparatus 1 to which the present invention is applied may be able to lock the rotation of the camera 21 at a specific angle, and if the main body 11 is gripped and stabilized, the camera 21 is also stabilized.

Also, in recent years, in the event of an accident, a drive mounted on a car or bicycle, which aims to contribute to the objective verification of the accident by recording video data and data by GPS (Global Positioning System). Recorders are spreading rapidly.
It is possible to use it as a drive recorder by installing application software on a smartphone, but it is not widely used.
The reason for the case of a car is that if you use a holder on the dashboard of the car to fix the smartphone etc. with a sucker or double-sided tape and fix the smartphone so that the driver can easily see the display part, the optical axis of the camera of the smartphone The direction is not the front or the back of the car you want to shoot, and the dashboard is reflected.
In other words, if the smartphone is placed in a position that is easy for the driver to see and does not interfere with driving, it cannot play the role of a drive recorder. On the other hand, if priority is given to the image as a drive recorder, the smartphone must be installed in a position that is difficult for the driver to see. By doing so, it becomes impossible to satisfy the dual purpose such as wanting to use the smartphone as a drive recorder and a navigation system. The same applies to bicycles. Therefore, it is considered that the use of smartphones as drive recorders is not widely used.
In other words, there is a problem that while using a smartphone as a drive recorder, we want to use it for other purposes at the same time.
The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, in the information processing apparatus 1 to which the present invention is applied, the camera 21 can be directed in any direction with respect to the apparatus main body, and the camera viewpoint image is displayed on a part of the screen while displaying the navigation map. Since it can be displayed in the form of a Picture in Picture, it can be used for other purposes at the same time while using the smartphone as a drive recorder. Therefore, a dedicated drive recorder becomes unnecessary.
The camera 21 can be rotated on two axes, such as installing it on a car so that it has a horizontally long screen, but installing it on a bicycle so that it has a vertically long screen, so it can be installed in a wide range of ways. Is possible.

In the future, in the smartphone industry, it is expected that there will be an increasing number of cases where multiple sensors are mounted on the rear camera and used not only for taking pictures but also for measuring distances and zooming. However, if the same measures are taken on the front camera side, the cost will be doubled, so there is a high possibility that it will be forgotten, and there will be a difference in camera functions between the front and rear.

In other words, there is a problem that only the rear camera can be made highly functional. The solution to this problem is to adopt the information processing apparatus 1 to which the present invention is applied. That is, the information processing device 1 to which the present invention is applied can be solved if there is a rotating camera 21 assuming application to a stereo camera, and it is not necessary to add a function to the front camera. ..

With conventional information processing devices, it has not been easy to select images while checking them on a screen that displays captured images in full size.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 6, it is possible to provide a function of automatically reproducing an image taken by a camera in a slide show and sequentially selecting the images. It is an image in which three buttons, a delete button B1, a review button B2, and a hold button B3, are displayed.
That is, when the user selects any one button, the picture switches to the next picture.
The operation does not necessarily have to be done manually and may be automated by AI or the like. Since this is a function of application software, it can be realized even with a conventional information processing device.

With conventional information processing devices (for example, smartphones), when navigating indoors (A) R in stores, etc., it is necessary to take pictures while standing the smartphone vertically, and keeping the smartphone vertical while walking is not necessary. , It becomes an unnatural way of holding and I get tired.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 7B, the camera rotates with respect to an image taken by a 2-axis rotating camera, a 1-axis rotating camera, or a 360-degree camera. Since the camera can be turned in the direction of travel while the display is held at an easy-to-see angle, it is possible to provide a function for indoor navigation on the AR as shown in FIG. 7 (A) A. can.

For example, the guide information GI1 indicates on the AR that the amount of sugar is low (for example, the amount of storage in the user's house or the amount of display in the sales floor).
For example, the guide information GI2 indicates the direction of the discounted goods section on AR.
For example, the guide information GI3 indicates the display position of the 10% discount product on the AR.
For example, the guide information GI4 indicates the display position of the product of 10 color variations on the AR.
Further, the information processing device 1 performs navigation in the store, such as where the ideal to be purchased is located.
The information processing device 1 may generate a mark, sound, or vibration on the screen when the object that the user is looking for enters the camera viewpoint image on the camera.
The information processing apparatus 1 may notify the user by lowering or raising the volume when the object is moved away from the object.
Further, a conventional information processing device having a fixed angle of the camera may be equipped with the same function.

In a conventional information processing device (for example, a smartphone), when navigating by a pedestrian (A) R outdoors, it is necessary to take a picture of the surroundings while standing the smartphone vertically.
Therefore, keeping the smartphone vertical while walking is an unnatural way to hold it and makes you tired. Therefore, navigation by AR is not common and is rarely used. Therefore, advertisements are rarely placed in the pedestrian (A) R navigation in consideration of the position and the direction of the camera.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 8B, it is easy to see the display for an image taken by a 2-axis rotating camera, a 1-axis rotating camera, or a 360-degree camera. Since the camera can be pointed in the direction of travel while being held at an angle, navigation by AR can be performed as shown in FIG. 8 (A) in a state where it is easy to see and does not get tired easily. Therefore, it may be possible to provide a function of displaying an image or a voice advertisement on the (A) R.

For example, the advertisement AD1 shows an advertisement with location information indicating that a 10% discount sale for all products is being carried out on AR.
For example, the advertisement AD2 shows an advertisement with location information of a store name of a store that will start business soon or a store that has just opened, on AR.
Here, the information processing apparatus 1 may allow the user to select in the menu whether or not to display the AR advertisement.
Further, the information processing apparatus 1 may be able to select from the menu, for example, the content is limited to sale information, restaurant advertisements, and the like.
In addition, the advertiser side may also be able to specify the attributes and the like of the advertisement target user in the menu.
Further, the information processing apparatus 1 may notify the user by sound or vibration when the advertisement is displayed.
Further, the information processing apparatus 1 may have a function of automatically rotating the camera toward the place where the advertisement is displayed so as to notice the advertisement.
Further, a conventional information processing device having a fixed angle of the camera may be equipped with the same function.

Conventional information processing devices (for example, smartphones) also have a function of displaying translations in AR for images taken by a camera. However, as mentioned above, AR navigation is not very common because it is necessary to hold the smartphone in an unnatural posture, and in that, translation etc. with AR without switching application software from AR navigation etc. It was not easy to issue.
According to one embodiment of the information processing apparatus 1 of the present invention, an image taken by a 2-axis rotating camera, a 1-axis rotating camera, or a 360-degree camera is translated on AR (during navigation, etc.). It is possible to provide a function to output information.

The Street View service started by Google LLC only displays images taken in the past, so it was not easy to display each point on the map in near real time.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 9, a background used in AR navigation taken by a 2-axis rotating camera, a 1-axis rotating camera, or a 360-degree camera while driving a car. By uploading the video data to the cloud CD, it provides a function to display the street view in near real time by displaying the background video that passed through the point on the map of the street view with a navigation service etc. can do.
Here, in the near real-time street view, not only the absolute time but also the relative time, that is, the information of how many minutes ago may be displayed.
Further, the server or the like connected to the cloud CD may create an image by combining the images of a plurality of cameras.
Further, when the information processing apparatus 1 specifies a certain point, the image passing in front of the point may be displayed in chronological order from the latest one.
By doing so, for example, the user can browse the video RM of the procession of the ramen shop in advance on a mobile information terminal, check the number of people lined up in front of the famous ramen shop, or the video of the gas station. You can use it to see the price information of GS.
In addition to the above, if a certain point is specified, the user may be notified each time an image of that place is taken after that.
Further, the server or the like connected to the cloud CD may create a database that can list the prices of gas stations in real time from the above data.

In the conventional information processing device, the main body and the camera are fixed, and the direction of the main body and the direction of the camera are the same, but in the case of the information processing device 1 in which the camera rotates, the directions of the main body and the camera are different. In many cases, the information processing apparatus 1 needs a mechanism for determining the direction of each.
Therefore, in one embodiment of the information processing apparatus 1 of the present invention, as shown in FIGS. 10 or 11, a display that displays an image pickup unit that captures a subject and an image pickup image including the subject imaged by the image pickup unit. The image pickup unit is arranged so as to change the angle between the main body portion having a larger volume than the image pickup unit and the optical axis direction of the image pickup unit and the normal direction of the display unit. , A hinge portion that supports the display unit so as to be relatively rotatably supported with respect to the main body portion while supporting both holding shafts with a rotation axis parallel to the lateral direction of the display unit, and the other that faces the one surface of the main body portion. A recess provided on one surface and capable of accommodating the image pickup portion rotated to the other one surface side by the hinge portion so as to overlap the main body portion in the thickness direction, and the main body is arranged on the main body portion as a single unit. It includes a first sensor that detects the direction of the unit, and a second sensor that is arranged in the image pickup unit and detects the direction of the image pickup unit by itself. Then, it is configured to add direction information to the digital photo file based on the direction of the image pickup unit detected by the second sensor.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 10 or 11, the Hall element switches the screen according to the relative angle with the screen of the camera, and the accelerometer is used to capture a photograph. Add direction information.
FIG. 10 is a diagram showing a state in which the information processing apparatus 1 is in a horizontal position, and information on a direction at the time of imaging by a camera directed in a direction different from that of the main body is added to a photograph, and a display method is shown.
FIG. 10A shows a usage state in the horizontal position, and shows a common main body posture when the camera 21 is rotated around the rotation shaft Z1 and the rotation shaft Z3. That is, the display unit 13 is substantially vertical and faces the user side, indicating that the subject exists on the back side of the display unit 13 in a positional relationship that can be captured by a rear camera.
FIG. 10B shows a usage state similar to that of a normal rear camera in the horizontal position, and the same vertical (top and bottom) information as the main body is added to the photograph.
FIG. 10C shows a state in which the camera 21 is rotated 90 degrees around the rotation axis Z1 in the horizontal position. In such a state, the information processing apparatus 1 does not invert the display up and down, but is a switching point where the display is inverted left and right if the mirror mode is ON for the left and right. The mirror mode is a mode in which an image captured by a camera displayed on a display is flipped horizontally and displayed like a mirror to take a picture. Normally, the information processing device is often turned on. Of course, it is often possible for the user to set it. That is, by rotating the camera 21 around the rotation axis Z1 by about 90 degrees or more or more than about 90 degrees, the left and right sides are reversed when the mirror mode is ON.
FIG. 10D shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 in the horizontal position. In such a state, the top and bottom are not inverted, but if the mirror mode is ON, the left and right are inverted.
FIG. 10E shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 and the camera 21 is rotated 45 degrees around the rotation axis Z3 in the horizontal position. In such a state, the top and bottom are not inverted, but if the mirror mode is ON, the left and right are inverted.
FIG. 10F shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 and the camera 21 is rotated 90 degrees around the rotation axis Z3 in the horizontal position. In such a state, inversion is a switching point for up and down, and at the same time, it is a display switching point for left and right if the mirror mode for inversion is ON.
FIG. 10 (G) shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 and the camera 21 is rotated 180 degrees around the rotation axis Z3 in the horizontal position. In such a state, the top and bottom are inverted, and the left and right are switched between the stage (C) and the stage (F), and since they have returned to their original positions, they are not inverted.

FIG. 11 shows a state in which the information processing device of FIG. 1 is in a vertical position, and information on the top and bottom (top and bottom) at the time of imaging by a camera oriented in a direction different from that of the main body is added to the photograph, and how to display the photograph. It is a figure which shows.
FIG. 11A shows a usage state in a vertical position, and shows a common main body posture when the camera 21 is rotated around the rotation shaft Z1 and the rotation shaft Z3. That is, the display unit 13 is substantially vertical and faces the user side, indicating that the subject exists on the back side of the display unit 13 in a positional relationship that can be captured by a rear camera.
FIG. 11B shows a state of use similar to that of a normal rear camera in a vertical position, and the same vertical (top and bottom) information as that of the main body is added to the photograph.
FIG. 11C shows a state in which the camera 21 is rotated 90 degrees around the rotation axis Z1 in the vertical position. In such a state, it is a switching point for flipping up and down, and at the same time, it is a switching point for displaying when the mirror mode for flipping left and right is ON.
FIG. 11D shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 in the vertical position. In such a state, the top and bottom are inverted, and if the mirror mode is ON, the left and right are also inverted.
FIG. 11E shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 and the camera 21 is rotated 45 degrees around the rotation axis Z3 in the vertical position. In such a state, the top and bottom are inverted, and if the mirror mode is ON, the left and right are also inverted.
FIG. 11F shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 and the camera 21 is rotated 90 degrees around the rotation axis Z3 in the vertical position. In such a state, the top and bottom are reversed. For left and right, if the mirror mode is ON, it is a switching point to invert.
FIG. 11G shows a state in which the camera 21 is rotated 180 degrees around the rotation axis Z1 and the camera 21 is rotated 180 degrees around the rotation axis Z3 in the vertical position. In such a state, the top and bottom are inverted, and the left and right are switched between the stage (C) and the stage (F) for the left and right, and since they have returned to their original positions, they are not inverted.

Some conventional information processing devices, such as Samsung's product "Galaxy note 9", have a shutter remote control that can be removed from the main body, but the shutter button is attached to the main body. I couldn't press it.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 12, the shutter remote controller RC is removable from the main body of the information processing apparatus 1.
FIG. 12A shows a shutter remote controller RC attached to the main body.
By fixing the shutter remote controller RC to the side of the main body, the shutter button provided on the shutter remote controller RC can be pressed even when the main body is attached. The portion beyond the button of the shutter remote controller RC may be stored inside the main body. Further, the shutter remote controller RC may be supplied with electric power from the main body when the main body is attached. Further, when the shutter remote controller RC is attached to the main body, the shutter remote controller RC may communicate with the main body in a contact manner. As a result, in addition to the buttons displayed on the display, a physical shutter button can be provided, which improves usability and operability.
FIG. 12B shows the shutter remote controller RC in a state of being removed from the main body.
That is, the shutter can be released by pressing a button even when the main body is attached, and the shutter can be released remotely by storing electricity in a battery or a capacitor even if the shutter is removed from the main body.
Further, the shutter remote controller RC may be equipped with a microphone so that commands can be remotely sent to the information processing apparatus 1 by voice.

FIG. 13 shows an automatic translation of a book and reading of receipt data by setting a mirror ML in front of a front camera of an information processing apparatus SP equipped with a fixed camera, which is different from the information processing apparatus of FIG. It is a figure which shows the state.
FIG. 13A shows an example in which a mirror ML is set in front of the front camera and the book is automatically translated.
FIG. 13B shows an example of a reflection image by an actual mirror ML.
FIG. 13C shows an example of the imageable range when the mirror ML is set in front of the front camera of the information processing apparatus SP provided with the fixed camera mounted on the base BS.
First, the mirror ML is attached to the front camera of the information processing device SP, the automatic translation reading aloud application of the book of the information processing device SP is started, and the language of the translation destination is specified. The initial setting may be in the language of the OS, and this specification may be omitted from the next time onward. Take a picture of the title of the book reflected in the mirror ML with the front camera with the book translation + reading aloud application.
However, when shooting with the front camera, if the mirror ML is attached, the image captured by the Pront camera will be a mirror image, and if the mirror mode is ON, it will be a mirror image of the mirror image, so it will return to the correct direction. The mode needs to be turned on.
Then, if necessary, the captured image is corrected. For example, rotate the book, correct the curvature of the book, correct the trapezoidal shape of the book, adjust the contrast of the book, cut out only the book part, delete the reflected finger, etc. to convert the characters into a state that is easy to recognize. By multiplying OCR (Optical Character Recognition / Reader) from, the text is recognized and extracted as a sentence, not one letter, number or word. Identify the language described in the book from the content of the text. At the time of these processes, image recognition by AI and sentence analysis by AI are performed, and the text misread by being converted into text by OCR is automatically corrected or manually input by the user. The text data is transmitted to the translation service of the cloud CD, recognizes the original language, translates into the language being set, and instantly transmits the bilingual data to the information processing apparatus SP.

In this embodiment, the translation is performed using the translation service of the cloud CD, but in addition to this, the information processing apparatus SP may be used alone for translation. If necessary, read aloud data is created from the translated text data. The reading aloud data may be created by a cloud CD, or may be created by the information processing apparatus SP alone.
After that, both the text in the language of the original book and the text in the translated language are displayed on the display unit of the information processing apparatus SP, and the reading data is reproduced by the speaker of the information processing apparatus SP.
In addition to the text data of the original book and its bilingual data, the image currently being captured by the front camera may be displayed in a small size. As a result, the user can grasp information such as the book being displaced from the front camera in almost real time. If the user wants to modify the displayed text, the text can be modified by the modification function and then manually input. This will interrupt the translation operation.
When the title of a book is confirmed, the user searches for the same title in the existing book database or reference data that has already been converted into an ebook, and if the previous translation end position is known, the user can use it. May be communicated through the information processing device SP.

Next, the translation process of the contents of the book will be explained.
It is almost the same as the translation process of the title mentioned above, but if there is a sentence that is not completed on the previous page, it is different that it is completed as one sentence together with the sentence on the current page and then translated. ..
Furthermore, if there is the same data as the past database or reference data, it can be considered as an additional process to bring the past data or reference data.
If you still have the pages you want to translate, you can turn the pages of the book or shoot the next page next to it to start the translation process again.
In the present invention, each page of a book is photographed as a still image without pressing the shutter, and the page is turned or the captured image is significantly changed. For example, the user captures a page captured by a camera. When the page on the left is intentionally shifted to the page on the right, this is detected and the page is reloaded, and the above-mentioned series of translation processes are repeated to perform translation.
The page may be reloaded by voice or by sound. Furthermore, the page may be reloaded using the buttons on the screen or the remote controller.
Alternatively, you may translate part of it, skip the translation process, and use the translated e-booked reference data. You may perform translation once, remember the page and its position in the page when the translation is stopped, and notify the user by displaying or reading aloud the page where the translation was stopped the next time when the translation is restarted.
The reading aloud process is almost the same as the translation process, and the data converted into text by OCR is read aloud without translation. The reading aloud process is the translation process with only the parts to be translated removed.
Not limited to books, the process of converting paper media into text is almost the same, skipping the process of translation and reading aloud, and displaying the text. In addition to displaying, for data such as receipts, understand the input items and automatically enter numbers and text in the columns as appropriate items. For example, the amount, date, address, etc. of the receipt can be automatically entered correctly.
In addition to the smart-type information processing device, a mirror ML is attached to the front camera of a notebook computer (hereinafter referred to as "personal computer") to read paper data such as user books and receipts on the desk. Is also possible.
In the case of a mirror attached to the front camera of a personal computer, the camera is often mounted on the display of the personal computer, and the angle of the display can be freely changed by the user, so the rotation mechanism that can freely change the angle of the mirror. May be installed.
By applying OCR to photographed books, receipts, etc., text data can be recognized and input. The user hardly needs to operate the information processing apparatus 1, and only needs to concentrate on arranging receipts and the like, and the operation can be largely omitted. Especially if you do it on a flat surface like a desk, it is easy to arrange receipts.
Furthermore, the mirror ML attached to the information processing device SP can be taken with a wider angle of view by using a wide-angle mirror such as a convex mirror instead of an ordinary flat mirror. You can save the trouble of moving it from side to side. There is also an advantage that the height of the stand on which the information processing apparatus SP is placed can be lowered. However, not only such a merit but also a demerit is that a large amount of distortion is taken, so it may be necessary to correct the captured image.

With a conventional information processing device (for example, a smartphone), when enjoying an AR game while walking outdoors, it is necessary to take a picture of the surroundings while standing the smartphone vertically, and keep the smartphone vertical while walking. Becomes unnatural and tired. Therefore, walking games using AR are not common and are rarely used. Therefore, the pedestrian (A) R game is not often played.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 8, the display is easily seen at an angle with respect to an image taken by a 2-axis rotating camera, a 1-axis rotating camera, or a 360-degree camera. Since the camera can be pointed in the direction of travel while holding it, an AR game can be played. Therefore, it is possible to cooperate with or fight with people who are participating at the same time.
That is, in the example shown in FIG. 14, the game image G1 of AR, which is supposed to capture one game character GC from behind, is displayed on the information processing device 1-1 according to the positions of the two players. Then, the information processing apparatus 1-2 displays the AR game image G2, which is supposed to be captured from the front.

In the conventional information processing device, since the camera does not rotate electrically, it is not easy to track and shoot the sound source SS.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 15, a sound is produced by using a plurality of microphones while shooting with a 2-axis rotating camera and a 1-axis rotating camera. It is a function that electrically directs the camera in the direction in which it is facing. That is, it is possible to provide a function of rotating the camera 21 in conjunction with sound.
It is preferable that the information processing apparatus 1 is placed on a desk or the like so that the user can confirm to some extent that he / she can take a picture by himself / herself.

With conventional information processing devices, it has not been easy to utilize the problems described in paper media for education. Taking a picture of paper with a camera requires a certain height, and in a conventional information processing device, in order to take a picture of paper with a front camera, it is necessary to use a mirror as described above or rotate the camera. Because.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 16, when a problem written on paper or the like is photographed by a 2-axis rotating camera, a 1-axis rotating camera, or a 360-degree camera, the problem is automatically taken. , AI is given a hint of the problem, for example, drawing an auxiliary line AL of the graphic problem.
FIG. 16A shows a state in which the question paper QP is imaged using the information processing apparatus 1.
FIG. 16B shows that the problem PQ to be processed has been recognized by the information processing apparatus 1 from the problem described in the problem sheet QP.
FIG. 16C shows an example of a problem image QG in which the AI included in the information processing apparatus 1 gives a hint, for example, an auxiliary line AL of a graphic problem, to the problem PQ.
A mirror as shown in FIG. 13 may be attached to the front camera to photograph such a problem and give a hint of the problem.

With conventional information processing equipment, it has not been easy to self-stabilize with a foldable stand.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 17, the foldable stand 14 is arranged in consideration of the center of gravity of the smartphone body.
FIG. 17A is a diagram showing a region surrounded by three points where the information processing apparatus 1 main body and the foldable stand 14 are in contact with each other when the information processing apparatus 1 is made independent.
That is, when the information processing device 1 is placed on a desk, for example, a triangle is drawn on the surface of the desk. It is shown that there are three vertices, the vertex VB of the information processing device 1 and the vertex VC of the main body angle of the information processing device 1.
FIG. 17B shows the positional relationship between the position of the center of gravity of the information processing apparatus 1 main body and the area surrounded by the three points to be grounded when the information processing apparatus 1 is made independent, when viewed from directly above. It is a figure which shows.
That is, the point directly below the position of the center of gravity G of the information processing apparatus 1 is within the region surrounded by the three vertices VA, VB, and VC that are in contact with the ground, and from any three sides that are straight lines connecting these two points. The foldable stand 14 is arranged in consideration of being far away and lowering the center of gravity G of the information processing apparatus 1 main body.
By doing so, when the information processing apparatus 1 is made independent, it becomes possible to make it more stable.
If the contact point between the stand and the desk is not one as shown in FIG. 17 (A) but a plurality of contacts, for example, two, the apex VB of the main body angle of the information processing apparatus 1 and the information processing apparatus. If the information processing device 1 has the center of gravity G of the information processing device 1 when viewed from directly above in a square composed of four points including the two points of the apex VC of the main body angle of 1, the information processing device 1 is installed. When it is made independent, it becomes possible to make it more stable.

The conventional information processing device does not have a function of searching for a photograph taken at the same or close position from the photographs taken in the past based on the information of the position where the photograph is to be taken.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 27, the existence of a past photograph taken at the same location is taught and the photograph is displayed. By doing so, the user can take the same pose as the photograph taken at this place with the children in the past, for example, and the information processing apparatus 1 can display them side by side in comparison.
In this case, the information processing apparatus 1 may limit the function by setting the home or the like as a non-display area. By doing so, the information processing apparatus 1 can prevent a large number of photographs from being applied.
FIG. 18A shows an example of a message MS that teaches the existence of past photo OPs taken at the same location.
FIG. 18B shows an example of a small window display of a past photograph OP taken at the same location and a display of a semi-transparent button HB that superimposes and displays an image captured by a camera in a semi-transparent manner.
FIG. 18C shows an example of a superposed display of a semi-transparent image of a past photograph OP taken at the same location and an image captured by a camera.

With conventional information processing devices, when searching for unknown words or meal names, there is no choice but to search from text or photo information, and it is easy to find the corresponding photo on the WEB based on the searched location. It wasn't.
According to one embodiment of the information processing apparatus 1 of the present invention, as shown in FIG. 28, for example, when a restaurant menu MN is photographed and the dish name OW in the photographed image is traced, only the translated text TW is displayed. Instead, it provides application software that displays the corresponding food image photograph IM taken in the same area from the search position information of GPS or Wi-Fi (registered trademark). This enables highly accurate photo search.
Further, when the dish name OW is derived from the place name, the information processing apparatus 1 displays the map MP and the place name PW displaying the corresponding points.

FIG. 20 is a diagram showing each state in which the shape is changed when the information processing apparatus 1 is provided with a sliding rotary camera.
That is, the main body is from a slide movable portion consisting of at least two parts, a front side including the display unit 13 and a back side of the main body portion 11 including the camera base portion 12-1 of the rotating camera 21 and the camera pointing portion 12-2. Each state in the configured information processing apparatus 1 is shown.

FIG. 20A is a diagram showing a state in which two slide movable parts are overlapped to the maximum extent.
FIG. 20B is a diagram showing a state in which the two slide movable parts are slid to the maximum. In the figure, reference numeral P indicates the rotation direction of the camera-oriented unit 12-2 (rotating camera 21).
FIG. 20C is a diagram showing a state in which the rotating camera 21 is rotated by sliding the two sliding movable portions to the maximum extent. By relatively sliding the display unit 13 and the main body unit 11, the entire camera unit including the camera base unit 12-1 and the camera-oriented unit 12-2 can be rotated by 180 degrees or more. The relative slide means that the display unit 13 is pushed by the main body portion 11 with a finger and slid, and vice versa.

The information processing apparatus 1 including the slide-type rotating camera includes an image pickup unit that captures a subject, a display unit that overlaps the image pickup unit in the thickness direction and displays an image captured image including the subject imaged by the image pickup unit, and the above-mentioned. While supporting the image pickup unit with a first rotation axis parallel to the lateral direction of the display unit so as to change the angle formed by the optical axis direction of the image pickup unit and the normal direction with respect to the display unit. A hinge portion that rotatably supports the main body portion is provided.
The image pickup unit can be translated relative to the display unit from a position where the first rotation axis overlaps with the display unit in the thickness direction to a position where the first rotation axis does not overlap with the display unit.
In the case of the information processing apparatus 1, when the slide movable portion is slid to the maximum, the rotation camera 21 has a range of motion from 0 ° to 270 ° and further.
Note that 0 ° is a state in which the rotating camera 21 is housed in a groove or a notch portion of the main body portion 11 (a state in which the rotating camera 21 and the main body portion 11 overlap in the thickness direction). 180 ° is a state in which the rotating cameras 21 are linearly arranged outside the main body 11. When it exceeds 180 °, the rotating camera 21 protrudes toward the front surface 11a of the main body portion 11, and at 270 °, the optical axis of the rotating camera 21 faces directly below the main body portion 11.
By doing so, the information processing apparatus 1 can be designed so as not to make a notch for the rotating camera while maximizing the screen occupancy of the display unit 13. In addition, the camera may automatically rotate when the display unit 13 is slid by a manual operation such as by hand or an electric operation such as pressing a button. This may be done by a torsion coil spring or electric rotation by a motor.

FIG. 21 is a diagram showing the structure of the biaxial rotation mechanism portion of the information processing apparatus 1 of FIG.
FIG. 21 (A) is a diagram showing a housing of the biaxial rotation mechanism unit.
FIG. 21B is a diagram showing a case where a biaxial hinge is used as the biaxial rotation mechanism portion.

FIG. 21C is a diagram showing a configuration example in which a hinge mechanism including a rotation shaft is provided on the housing itself as a two-axis rotation mechanism portion.

As shown in FIGS. 20A to 21C, the information processing apparatus 1 includes an image pickup unit that captures an image of a subject, the subject that overlaps the image pickup unit in the thickness direction, and is imaged by the image pickup unit. The first image pickup unit is parallel to the short side of the display unit so as to change the angle formed by the display unit that displays the captured image and the optical axis direction of the image pickup unit and the normal direction with respect to the display unit. It is provided with a hinge portion that rotatably supports the main body portion while supporting both holding shafts with the rotating shaft of the above.
The image pickup unit can be translated relative to the display unit from a position where the first rotation axis overlaps with the display unit in the thickness direction to a position where the first rotation axis does not overlap with the display unit.
The image pickup unit is a drive unit that drives the hinge unit to rotate the image pickup unit, and a control unit that controls the drive unit so as to rotate the image pickup unit in an instruction direction in response to a movement instruction to the image pickup unit. And include.
The image pickup unit can be rotated by a second rotation axis parallel to the longitudinal direction of the display unit.
The image pickup unit can rotate on the second rotation axis at a position that does not overlap with the display unit in the thickness direction.
The display unit includes a touch panel for performing the movement instruction by a touch operation.
The hinge portion includes a first rotation shaft arranged parallel to the lateral direction of the display portion and a second rotation shaft arranged parallel to the longitudinal direction of the display portion. The first rotation axis is configured so as not to rotate in accordance with the rotation of the image pickup unit when the image pickup unit is rotated about the second rotation axis.

A motor for driving the two-axis rotation mechanism unit, a controller for controlling the motor in response to a user's screen operation or remote operation, and the like shall be provided in the housing. Instead of providing a special controller, the CPU of the information processing apparatus 1 may control the controller.
It is conceivable to realize wiring such as providing a cavity penetrating in the axial direction inside the rotating shafts HS1 and HS3 and passing the FPC through the cavity, or winding the FPC around the rotating shafts HS1 and HS3.
By doing so, the information processing apparatus 1 can reduce the size of the biaxial rotation mechanism unit as compared with the case of using the biaxial hinge.

FIG. 22 is a diagram showing each state in which the shape is changed when a rotating camera is provided in the notch portion of the display unit of the information processing apparatus of FIG. 1.
FIG. 22A is a diagram showing a state in which the camera is a rear camera.
FIG. 22B is a diagram showing a state in which the camera is rotated.
The information processing apparatus 1 of FIG. 22 is provided with a notch (what is called a notch) in a part of the display unit 13, and the notch portion is provided with a rotary camera.
By doing so, it is possible to rotate the camera base 12-1 and the camera-oriented unit 12-2 by 180 degrees or more while maximizing the screen occupancy ratio of the display unit 13 with respect to the main body.

In general, in order to read a book written in a language that you do not understand, you need to learn that language and understand its meaning, or wait until it is translated into your native language. There was a big hurdle for those who wanted to read. Not only those who want to read books in foreign languages in Japan, but also foreigners have the same hurdles. For example, prints distributed at foreign children's schools for foreigners living in Japan, for example, when Japanese manga is not immediately translated into their native language, or foreigners in Japan do not understand Japanese guidebooks. I often hear complaints that I don't understand the meaning of such things.
As one of the methods for resolving such dissatisfaction, it is conceivable to translate in the form shown in FIG.
Hereinafter, the information processing apparatus SP having a translation function and a reading aloud function in addition to the translation function will be described.
(Translation function + reading aloud function)
The translation and reading aloud functions performed by the information processing apparatus SP in the state shown in FIG. 13 and the system including the cloud CD (not shown) will be described in detail. The cloud CD is a server equipped with an AI translation function, a reading aloud data generation function, and the like.
In this system, the information processing apparatus SP and the cloud CD are connected via a network. Information is exchanged between the information processing device SP and the cloud CD. For example, the information processing apparatus SP transmits a request for translation and reading aloud processing including an image captured by a front camera to a cloud CD, and receives a response including text data and reading aloud data as a result of processing the request with the cloud CD. do.

The information processing apparatus SP includes a wireless communication unit, a CPU, a memory, a display unit, a speaker, a microphone, an image pickup unit including a camera, and the like in the device main body. The wireless communication unit communicates with other information processing devices (other smartphones, cloud CD servers, etc.) through a mobile communication network, wireless LAN, or the like. Software for application software (hereinafter referred to as "automatic translation reading application") that executes translation and reading aloud processing is installed in the memory, and the CPU reads the automatic translation reading application in the memory to perform translation and reading aloud. Realize the function.
The CPU loaded with the automatic translation aloud application transmits the captured image obtained by shooting a book to the cloud CD via a mirror ML separate from the main body of the device, which refracts the optical axis direction of the image pickup unit. The cloud CD functions as a control means for receiving the text data corresponding to the character string generated based on the captured image and displaying it on the display unit of the main body of the apparatus.

That is, the automatic translation reading aloud application is used in an information processing apparatus SP including a camera in which a sentence is described but a book is imaged, and a display unit capable of displaying an image captured by the camera, text data, and the like. The text may be written vertically.

In the automatic translation reading app, the image pickup unit mounted on the device body causes the image pickup unit to refrain the optical axis direction of the image pickup unit, and the recording medium on which the character string is arranged is photographed via a mirror ML separate from the device body. A step of transmitting the captured image to the cloud CD, a step of generating text data corresponding to the character string based on the captured image, and a step of displaying the text data on the display unit of the information processing device are executed. It is a computer program.
Specifically, the automatic translation reading app causes the camera of the image pickup unit to take a picture of a book in which sentences are written from the direction of the optical axis refracted through the mirror ML attached to the main body of the information processing device, and takes an image. The photograph is transmitted to a cloud CD, text data corresponding to a sentence generated based on the captured image is received by the cloud CD, and the received text data and the captured image captured by the camera of the imaging unit in near real time. It has a function to display a UI screen that presents both and on the display unit. The text may be written vertically or horizontally.
Further, this automatic translation reading aloud application has an image correction function of correcting an captured image into an image suitable for generating text data on a cloud CD. The automatic translation reading aloud application has a processing function for performing at least one of translation and reading aloud of text data.
Further, the automatic translation reading aloud application has a combining function of combining with other parts of the captured image or text data obtained from the next captured image to complete one sentence when the text data is incomplete as a sentence.

In the above embodiment, the automatic translation reading application is installed in the information processing device SP shown in FIG. 13 to realize functions such as translation and reading aloud. In addition, the automatic translation reading application is installed in the information processing device 1. It may be installed to realize functions such as translation and reading aloud.
In this case, the information processing apparatus 1 has a rotating camera 21 for capturing a book in which sentences are described, and a display unit capable of displaying an captured image, text data, etc. captured by the rotating camera 21, as shown in FIGS. 20 to 22. 13 and. The rotating camera 21 of the information processing device 1 is rotatably supported by a rotating mechanism so that the optical axis faces the front side on which text data is presented from the back side of the main body 11 of the information processing device 1.
The automatic translation aloud reading application installed in such an information processing device 1 corresponds to a character string generated based on a captured image by the cloud by sending a photograph of a book containing sentences to the cloud. It has a presentation function that receives the text data to be processed and presents both the received text data and the captured image captured in near real time by the rotating camera.
Further, the presentation function of this automatic translation reading application is predetermined to be associated with the past text data or the barcode information from the text data of the medium on which the character string is arranged or the barcode information described in the medium. The reference data is presented with reference to the reference data.
The automatic translation aloud reading application flips left and right, flips vertically, rotates, corrects the curvature of the medium part where the character string is placed, corrects the trapezoidal part of the medium part where the character string is placed, and contrasts the captured image sent to the cloud. It has a correction function that corrects at least one of the adjustment of, cutting out only the medium part in which the character string is arranged, and deleting the image of the finger that has been reflected.

Here, the UI of the automatic translation reading aloud application will be described with reference to FIG. 23.
The automatic translation aloud application is activated by the user clicking an icon displayed on the screen of the information processing apparatus SP, and the user interface 31 shown in FIG. 23 (hereinafter referred to as "UI 31") is displayed on the screen of the display unit. Is displayed.

As shown in FIG. 23, the UI 31 is provided with a display area 32, a display area 33, a language designation unit 34, a captured image display area 35, correction buttons 36, 37, scroll bars S1, S2, and the like.
The display area 32 is an area in which the translation source text obtained by the character recognition process (OCR) of the captured image is folded back at the edge of the screen in the line direction (horizontal direction) and displayed. The display area 33 is an area in which the text obtained by translating (converting) the translation source text into the translation destination language is folded back at the edge of the screen in the line direction (horizontal direction) and displayed.
The language designation unit 34 is a part for designating the language of the translation source and the language of the translation destination, and is provided with a pull-down menu for designating the translation source language and a pull-down menu for designating the translation destination language.
The captured image display area 35 is an area in which the captured image captured by the front camera is displayed in real time, and the captured image is reduced and displayed.
When the correction button 36 is clicked, the text editing function of the automatic translation reading aloud application is activated, and the text in the display area 32 can be corrected. When the correction button 37 is clicked, the text editing function of the automatic translation reading aloud application is activated, and the text in the display area 33 can be corrected.
The scroll bar S1 is for scrolling and displaying the text of the line exceeding the display area 32 in the longitudinal direction of the screen by the user's operation. The scroll bar S2 is for scrolling and displaying the text of the line exceeding the display area 33 in the longitudinal direction of the screen by the user's operation.
In the automatic translation reading aloud application, each function is operated by selecting either translation or reading aloud from the UI 31 with the function selection button.
The cloud CD is, for example, a server that performs translation and reading aloud, and receives a request for image processing including an image from an automatic translation reading application of an information processing apparatus SP, performs translation processing and reading aloud, and responds to the request. The text data and the reading data of the translation result and the reading result are returned to the automatic translation reading application of the information processing apparatus SP.

Here, an operation example of translation and reading aloud will be described with reference to the flowcharts of FIGS. 24 and 25. 24 and 25 are flowchart examples showing the process of translation and reading aloud in a system including the information processing apparatus SP and the cloud CD.
In step S101 of FIG. 24, the mirror ML is attached to the main body of the apparatus so as to correspond to the front camera.
The user sets the book that opened the page to be read on the reading table (see FIG. 13C), clicks the icon of the information processing apparatus SP in step S102, and starts the automatic translation reading aloud application. The UI 31 is displayed on the display unit.
In step S103, when the user selects and specifies the translation source and translation destination languages from the pull-down menu of the language specification unit 34 of the UI 31, the automatic translation aloud reading application activates the front camera, and in step S104, the reading target is read by the front camera. An image of the page containing the title of the book is captured. The translation source language and the translation destination language may be set in advance as predetermined languages (for example, "Japanese", "English", etc.), and the translation source language is described in S107, which will be described later. It may be determined automatically. The image including the page captured by the front camera is reduced and displayed in the captured image display area 35 at the upper part of the UI 31 while maintaining the angle of view (aspect ratio). In the captured image display area 35, it is possible to check whether or not the pages of the book are deviated from the captured image area 35 of the front camera.

By confirming the captured image in the captured image display area 35, the user adjusts the position of the book or the position of the entire information processing apparatus SP so that the entire page to be read of the book is contained in the captured image display area 35, and the imaging range is adjusted. To adjust. When the imaging range is adjusted, the automatic translation reading aloud application transmits the captured image captured in the adjusted imaging range to the cloud CD.
In step S105, in the cloud CD, the server reverses left and right, flips vertically, rotates, corrects the curvature of the medium portion in which the character string is arranged, and characters with respect to the captured image received from the automatic translation reading application of the information processing apparatus SP. Image correction such as trapezoidal correction of the medium part where columns are arranged, contrast adjustment, cropping of only the medium part where character strings are arranged, deletion of the image of the finger that has been reflected, etc. are performed, and it is equivalent to one page of this book. Detects the width and the position of the character string.
In step S106, the server extracts text data (hereinafter referred to as "text") by recognizing characters from the captured image whose orientation and the like have been corrected. Character recognition is called OCR.
In step S107, the server determines the language of the characters described in the book from the extracted text.
In step S108, the server corrects the text based on the language of the determined characters, if necessary. “As needed” means, for example, a case where there are many characters that are easily misread or unrecognized characters and the character recognition rate is lower than a certain value. In addition, corrections are made such that the title and number of pages of books and chapters are not included in the text, and if ruby (reading) is added to the kanji, it will not be included in duplicate.
In step S109, the server translates the text into the designated language by the AI translation function.
In step S110, the server generates the read-aloud data from the translated text by the read-aloud data creation function, and transmits the processing result data to the information processing apparatus SP. The processing result data is the generated reading aloud data, the text before translation, the text after translation, and the like. The reading aloud data is obtained by adding attribute data (punctuation marks, accent data, ruby, etc.) to the text as a unit of the wording of the text for reading aloud.

When the automatic translation aloud reading application of the information processing apparatus SP receives the processing result data from the server, in step S111, the automatic translation aloud reading application displays the received processing result data on the UI 31 of the display unit and reads aloud.
In the UI 31, the text of the translation source language (original language) is displayed in the display area 32, and the text of the translated language (translation language) is displayed in the display area 33, respectively. At the same time, the reading aloud function of the automatic translation reading aloud application reads aloud the read aloud data of the received translated text and outputs the voice from the speaker.
Here, if the user finds an error in the text before or after translation and wants to correct it, in step S112, when the user clicks the correction button 36 or the correction button 37, the editing function is enabled and the correction button 36 is activated. Alternatively, since the text in the display area (display area 32 or display area 33) corresponding to the correction button 37 can be edited, the text in the display area is manually corrected (manually input) in step S113.

On the other hand, if there is no mistake in the text before or after translation, when the user clicks the confirmation button (not shown) of UI31, the cloud CD is notified that the translation has been confirmed. Even if the confirmation button is not pressed, it can be considered that there is no correction and the confirmation is made as long as the correction button is not pressed.
Upon receiving this notification, the cloud CD server extracts the title (title) of the book from the untranslated text in step S114.
Then, in step S115, the server selects the database of the published book, the translated electronic book-converted reference data stored in the memory in advance, and the like, based on the extracted book title (title) and the description language of the book. It matches with the contents (text) of the existing book by reference, identifies the title (title) of the existing book that matches the contents of the book, and notifies the information processing apparatus SP of it. If the title (title) of the existing book does not match as a result of matching, the automatic translation reading aloud application of the information processing apparatus SP is notified to that effect.

In the automatic translation reading app of the information processing device SP, when the server notifies that the title of the existing book matches, the automatic translation reading app reads the data including the text of the book based on the title of the existing book. If the end position of the page translated immediately before is held in step S116 by acquiring from the database or memory of the published book, the site of the book on the Internet, or the like, the text is displayed and read aloud from that position.
On the other hand, when the notification from the server that the title of the existing book does not match is received, the automatic translation reading aloud application captures the next page of the page translated immediately before in step S117, and captures the captured image. Send to cloud CD.

In the cloud CD, the server performs the same processing as in steps S105 to S108 in steps S118 to S121.
Then, in step S122, the server confirms the presence or absence of the text of the page that has not been completed as a sentence on the previous page (hereinafter, this is referred to as "unfinished sentence"), and if there is an unfinished sentence, the unfinished sentence. Concatenate the text and the newly read text to complete one sentence. In the example in the case of Japanese, at least the part of the character string constituting the text where the punctuation mark "." Is provided is concatenated.
Subsequently, in step S123, the server matches the existing book (published book, etc.) with the content of the book by referring to the published book database, reference data, etc., based on the sentence of one sentence and the description language of the book. , If the data match each other, that is, if they can be matched, the data including the text of the book is acquired.
After that, in step S124, the server translates the text into the designated language by the AI translation function as in step S109.
In step S125, the server generates reading data from the translated text by the reading data creation function, and transmits the processing result data to the automatic translation reading application of the information processing apparatus SP.

When the automatic translation aloud reading application of the information processing apparatus SP receives the processing result data from the server, in step S126, the automatic translation aloud reading application displays the received processing result data on the UI 31 of the display unit and reads aloud.
If there is an error in the text before and after translation displayed on the UI 31, the text is corrected in steps S127 and S128 in the same manner as in steps S112 and S113.
If there is no mistake in the text before and after the translation and there is the next translation page (Yes in step S129), the user turns the page of the book and has the turned page imaged.
Regarding the presence / absence of the next translation page, the presence / absence of an adjacent page of the captured image may be detected, or it may be detected whether or not a new page has been turned by the user.
Further, when the image of the book for two pages is included in the captured image, the automatic translation aloud reading application cuts out the image of the next next page from the captured image, sends it to the server, and proceeds to step S118.

If there is no next translation page in step S129 (No in step S129), a notification of the end of translation is transmitted to the cloud CD.
In the cloud CD, the server stores the translation end position of the text by the notification of the translation end received from the automatic translation aloud reading application of the information processing apparatus SP, and ends the translation and aloud processing on the server side in step S130.
According to the automatic translation reading app of the information processing device SP that operates in this way, when the information processing device SP with the mirror ML attached is set on the base BS and the book with the opened page is imaged with the front camera, the captured page is taken. Since the captured image of is displayed in the captured image display area 35 of the UI 31, the image of the book can be aligned while checking the captured image of the captured image display area 35, and the page of the book can be captured in the correct imaging range. Can be done. As a result, it is possible to eliminate page breaks and reduce mistakes when the information processing apparatus SP translates and reads aloud the text read from the book.
Further, according to the information processing device SP, the captured image of the book page is subjected to character recognition processing to extract the text, and the texts of the straddling pages are concatenated to convert the text into one sentence into the automatic translation reading application of the information processing device SP. Since it is captured and displayed in the display area 32 of the UI 31, it is possible to digitize a paper-based book in a short working time.
Furthermore, according to the automatic translation reading aloud application of the information processing apparatus SP, the title of the book is extracted from the spread pages of the book, the existing book is specified based on the text, and the data of the existing book is acquired. It is also possible to eliminate misreading and unreading as compared with translating and reading aloud the whole sentence by OCR, and it is possible to improve the translation accuracy of the book and to read aloud smoothly.

In the above operation example, the captured image display area 35 is provided on the upper part of the UI 31, but the captured image display area 35 is not limited to this example, and the text before or after translation is used somewhere in the UI 31. Should be displayed small.
Also, when the text is extracted, if the sentence is interrupted in the middle, the text that is interrupted scans the next page without translating or reading aloud on that page to make one sentence. After that, you may translate or read aloud.
Furthermore, if the book whose image was read was once scanned by someone and the text data of the book or data such as an electronic book is already stored in a memory or the like, the information in the memory should be used for translation and reading aloud. You may do it.
With this translation and reading function, each page of a book is taken as a still image without pressing the shutter, and the pages are turned or the captured image has changed significantly, for example, the page taken by the camera. When the user intentionally shifts from the left page to the right page, this is detected and the page is reloaded, and the above-mentioned series of translation processes are repeated to perform translation.
The page may be reloaded by voice or by sound. Furthermore, the page may be reloaded using the buttons on the screen or the remote controller.
Alternatively, you may translate part of it, skip the translation process, and use the translated e-booked reference data. You may perform translation once, remember the page and its position in the page when the translation is stopped, and notify the user by displaying or reading aloud the page where the translation was stopped the next time when the translation is restarted.

(Reading aloud function)
Next, the UI of the reading aloud function will be described with reference to FIG. 26. FIG. 26 is a diagram showing a UI of the reading aloud function. In FIG. 26, the same reference numerals are given to the same configurations as those in FIG. 23, and the description thereof will be omitted.
As shown in FIG. 26, the UI 31 of the reading aloud function is provided with a wide display area 32, and is not provided with the display area 33 of the translated text shown in FIG. 23. The language designation unit 34 is provided with only one pull-down menu for designating a text-based language.

Next, an operation example of the reading aloud function will be described with reference to the flowcharts of FIGS. 27 and 28. 27 and 28 are flowcharts showing a reading process in a system including an information processing apparatus SP and a cloud CD.
In step S201 of FIG. 27, the mirror ML is attached to the main body of the apparatus so as to correspond to the front camera.
The user sets the book that opened the page to be read on the reading table BS (see FIG. 13C), clicks the icon of the information processing apparatus SP in step S202, starts the automatic reading aloud application, and launches the automatic reading aloud application. As a result, the UI 31 is displayed on the display unit.
In step S203, when the user specifies a reading language from the pull-down menu of the language specification unit 34 of the UI 31, the automatic reading application activates the front camera, and in step S204, the page including the title of the book to be read by the front camera is displayed. The image is captured. As the reading aloud language, a predetermined language (for example, "Japanese" or "English") may be set as a default value in advance. In this case, it is not necessary to specify the reading aloud language with the default setting. The reading aloud language may be automatically determined by S207 described later.
The image including the page captured by the front camera is reduced and displayed in the captured image display area 35 at the upper part of the UI 31 while maintaining the angle of view (aspect ratio). In the captured image display area 35, it is possible to check whether or not the pages of the book are deviated from the captured image area 35 of the front camera.
By confirming the captured image in the captured image display area 35, the user adjusts the position of the book or the position of the entire information processing apparatus SP so that the entire page to be read of the book is contained in the captured image display area 35, and the imaging range is adjusted. To adjust. When the imaging range is adjusted, the automatic reading aloud application transmits the captured image captured in the adjusted imaging range to the cloud CD.

In step S205, in the cloud CD, the server reverses left and right, flips up and down, rotates, corrects the curvature of the medium portion in which the character string is arranged, and the character string with respect to the captured image received from the automatic reading application of the information processing apparatus SP. The width of one page is corrected by correcting the trapezoidal shape of the medium part where the character string is placed, adjusting the contrast, cutting out only the medium part where the character string is placed, and deleting the image of the finger that has been reflected. And detect the position of the character string.
In step S206, the server extracts text by recognizing characters from the captured image whose orientation and the like have been corrected. The process of recognizing characters from an image is called OCR.
In step S207, the server determines the language (description language) of the characters described in the book from the extracted text.
In step S208, the server corrects the text based on the language of the determined characters, if necessary. “As needed” means, for example, a case where there are many misread characters and unrecognized characters and the character recognition rate is lower than a certain value. In addition, corrections are made such that the title and number of pages of books and chapters are not included in the text, and if ruby (reading) is added to the kanji, it will not be included in duplicate.
In step S209, the server generates reading aloud data from the extracted text by the reading aloud data creation function, and transmits the processing result data to the information processing apparatus SP. The processing result data is the generated reading aloud data, text, and the like. The reading aloud data is obtained by adding attribute data (punctuation marks, accent data, ruby, etc.) to the text as a unit of the wording of the text for reading aloud.

When the automatic reading aloud application of the information processing apparatus SP receives the processing result data from the server, in step S210, the automatic reading aloud application displays the received processing result data on the UI 31 of the display unit and reads aloud.
In the UI 31, the text in the reading language is displayed in the display area 32. At the same time, the reading aloud function of the automatic reading aloud application reads aloud the read aloud data of the received translated text, and outputs the voice from the speaker of the information processing apparatus SP.
Here, if the user finds an error in the reading content or the text and wants to correct it, the user clicks the correction button 36 in step S211 to enable the editing function and edit the text in the display area 32. Therefore, in step S212, the text in the display area 32 is manually corrected (manually input). After modification, the text is sent to the server and reading data is generated in step 209.
On the other hand, if there is no mistake in the reading content or the text, when the user clicks the confirmation button (not shown) of the UI 31, the cloud CD is notified that the reading content has been confirmed. Even if the confirmation button is not pressed, it can be considered that there is no correction and the confirmation is made as long as the correction button is not pressed.

The server of the cloud CD that has received the notification from the automatic reading application of the information processing apparatus SP extracts the title (title) of the book from the text recognized by the characters in step S213.
Then, in step S214, the server refers to the database of the published book, the reference data stored in the memory in advance, and the like based on the title (title) of the extracted book and the description language of the book, and the contents of the existing book. Matches with (text), identifies the title (title) of an existing book that matches the content of the book, and notifies the automatic reading aloud application of the information processing apparatus SP. If the title (title) of the existing book does not match as a result of matching, the automatic reading aloud application of the information processing apparatus SP is notified to that effect.

In the automatic reading app of the information processing device SP, when there is a notification from the server that the title of the existing book matches, the automatic reading app publishes the book data based on the title of the existing book or the database of the book. If the end position of the page read aloud immediately before is held in step S215 after acquiring from a memory, a book site on the Internet, or the like, the text is displayed and read aloud from that position.
Further, when the notification from the server that the title of the existing book does not match is received, the automatic reading aloud application captures the next page of the page read aloud immediately before in step S216, and captures the captured image in the cloud. Send to CD.

In the cloud CD, the server performs the same processing as in steps S205 to S208 in steps S217 to S220.
Then, in step S221, the server confirms the presence or absence of the text of the page that has not been completed as a sentence on the previous page (hereinafter, this is referred to as "unfinished sentence"), and if there is an unfinished sentence, the unfinished sentence. Concatenate the text and the newly read text to complete one sentence. In the example in the case of Japanese, at least the part of the character string constituting the text where the punctuation mark "." Is provided is concatenated.

Subsequently, in step S222, the server matches the contents of the book with the existing book (published book, etc.) by referring to the database of the published book, reference data, etc., based on the sentence of one sentence and the description language of the book. Then, if the data match each other, that is, if they can be matched, the data including the text of the book is acquired.
After that, in step S223, the server generates the reading data from the text recognized by the character by the reading data creation function, and transmits the processing result data to the automatic reading application of the information processing apparatus SP, as in the step S209.

When the automatic reading aloud application of the information processing apparatus SP receives the processing result data from the server, in step S224, the automatic reading aloud application displays the received processing result data on the UI 31 of the display unit and reads aloud.
If there is an error in the text displayed on the UI 31, the text is corrected in steps S225 and S226 in the same manner as in steps S211 and S212.
If there is no mistake in the text read aloud and there is the next translation page (Yes in step S227), in step S228, the user turns the page of the book and captures the turned page in the automatic reading application of the information processing device SP. Let me. Regarding the presence / absence of the next reading aloud page, the presence / absence of an adjacent page of the captured image may be detected, or whether or not a new page has been turned by the user may be detected.
Further, when the image of the book for two pages is included in the captured image, the automatic reading aloud application cuts out the image of the next next page from the captured image, sends it to the server, and proceeds to step S217.
On the other hand, if there is no next reading aloud page in step S227 (No in step S227), a notification of the end of reading aloud is transmitted to the cloud CD.

In the cloud CD, the server stores the reading aloud data and the reading aloud end position of the text in step S229 by the notification of the reading aloud end received from the automatic reading aloud application of the information processing apparatus SP, and the reading process on the server side in step S230. To finish.

According to the automatic reading app of the information processing device SP that operates in this way, when the information processing device SP with the mirror ML attached is set on the base BS and the book whose page is opened is imaged by the front camera, the imaged page is displayed. Since the captured image is displayed in the captured image display area 35 of the UI 31, the image of the book can be aligned while checking the captured image of the captured image display area 35, and the page of the book can be captured in the correct imaging range. can. As a result, the page is not cut off, and the reading aloud error when the text read from the book is read aloud by the automatic reading aloud application of the information processing apparatus SP can be reduced.
Further, according to the automatic reading aloud application of the information processing apparatus SP, the captured image of the book page is subjected to character recognition processing to extract the text, and the texts of the straddling pages are concatenated and the text of one sentence is taken into the information processing apparatus SP. Since the information is displayed in the display area 32 of the UI 31, it is possible to digitize a paper-based book in a short working time.
Furthermore, according to the automatic reading aloud application of the information processing device SP, an existing book is identified based on the extracted text and the data of the existing book is acquired, so that the entire text is read aloud as compared with OCR. You can read aloud a book smoothly without reading aloud.

(Text conversion function)
Next, the text conversion function will be described. The UI of the text conversion function is basically the same as the UI 31 of FIG. 26, and the description thereof will be omitted.
Here, an operation example of the text conversion function will be described with reference to the flowcharts of FIGS. 29 and 30. 29 and 30 are flowcharts showing a text conversion process in a system including an information processing apparatus SP and a cloud CD.
In step S301 of FIG. 29, the mirror ML is attached to the main body of the apparatus so as to correspond to the front camera.
The user sets the book that opened the page to be read on the reading table (see FIG. 13C), clicks the icon of the text conversion function of the information processing apparatus SP in step S302, and starts the automatic text conversion application. As a result, the UI 31 is displayed on the display unit.
In step S303, when the user specifies a text conversion language from the pull-down menu of the language specification unit 34 of the UI 31, the automatic text conversion application activates the front camera, and in step S304, the title of the book to be read by the front camera is included. The image of the page is captured. As the text language, a predetermined language (for example, "Japanese" or "English") may be set as a default value in advance. In this case, it is not necessary to specify the text language with the default settings. .. The text language may be automatically determined by the process of step S307 described later.
The image including the page captured by the front camera is reduced and displayed in the captured image display area 35 at the upper part of the UI 31 while maintaining the angle of view (aspect ratio). In the captured image display area 35, it is possible to check whether or not the pages of the book are deviated from the captured image area 35 of the front camera.
By confirming the captured image in the captured image display area 35, the user adjusts the position of the book or the position of the entire information processing apparatus SP so that the entire page to be read of the book is contained in the captured image display area 35, and the imaging range is adjusted. To adjust. When the imaging range is adjusted, the automatic text conversion application transmits the captured image captured in the adjusted imaging range to the cloud CD.

In step S305, in the cloud CD, the server rotates the document, corrects the curvature, corrects the trapezoidal shape, adjusts the contrast, cuts out only this part, and reflects it from the captured image received from the automatic text conversion application of the information processing device SP. The image is corrected, such as deleting the image of the finger, and the width of one page and the position of the character string are detected.
In step S306, the server extracts text by recognizing characters from the captured image whose orientation and the like have been corrected. The process of recognizing characters from an image is called OCR.
In step S307, the server determines the language (description language) of the characters described in the book from the extracted text.
In step S308, the server corrects the text based on the language of the determined characters, if necessary. “As needed” means, for example, a case where there are many misread characters and unrecognized characters and the character recognition rate is lower than a certain value. In addition, corrections are made such that the title and number of pages of books and chapters are not included in the text, and if ruby (reading) is added to the kanji, it will not be included in duplicate.

When the text is received from the server in the automatic text conversion application of the information processing apparatus SP, in step S309, the automatic text conversion application displays the received processing result data on the UI 31 of the display unit.
In the UI 31, the text in the text language is displayed in the display area 32.
Here, if the user finds an error in the text and wants to correct it, in step S310, when the user clicks the correction button 36, the editing function is enabled and the text in the display area 32 can be edited. In S311 the text in the display area 32 is manually corrected (manually input). After modification, the text is sent to the server.
On the other hand, if there is no mistake in the text, when the user clicks the confirmation button (not shown) of the UI 31, the cloud CD is notified that the content of the text has been confirmed. Even if the confirmation button is not pressed, it can be considered that there is no correction and the confirmation is made as long as the correction button is not pressed.

In step S312, the server of the cloud CD notified by the automatic text conversion application of the information processing device SP has the title (title) of the book and the description of the book from the text modified by the information processing device SP or the text converted by the server. Extract the language.
Then, in step S313, the server refers to the database of the published book, the reference data stored in the memory in advance, and the contents of the existing book based on the title (title) of the extracted book and the description language of the book. Matches with (text), identifies the title (title) of an existing book that matches the content of the book, and notifies the information processing apparatus SP of the title (title) of the specified book and the description language. As a result of matching, if the title (title) of the existing book does not match, the automatic text conversion application of the information processing apparatus SP is notified to that effect.

In the automatic text conversion application of the information processing device SP, when the server notifies the existing book title (title) and the description language of the book, the automatic text conversion application uses the existing book title (title) and the book. If the book data is acquired from the published book database, memory, Internet book site, etc. based on the description language, and the end position of the page converted into text immediately before is held in step S314, from that position. Display text.
Further, in the automatic text conversion application of the information processing apparatus SP, when the notification from the server that the title of the existing book does not match is received, the automatic text conversion application advances the text conversion of the book, so that in step S315. , The next page of the page read aloud immediately before is imaged, and the captured image is transmitted to the cloud CD.

In the cloud CD, the server performs the same processing as in steps S305 to S308 in steps S316 to S319.
Then, in step S320, the server refers to the book of the published book, the reference data stored in the memory in advance, and the like based on the description language and the text of the book, and the book with the existing book (published book, etc.). If the data matches the contents of the book, that is, if the data can be matched, the data including the text of the book is acquired from the database or memory of the published book. After that, the server sends the acquired text to the automatic text conversion application of the information processing apparatus SP.

In the automatic text conversion application of the information processing apparatus SP, when the text is received from the server, the automatic text conversion application displays the text on the UI 31 of the display unit in step S321.
If there is an error in the text displayed on the UI 31, the text is corrected in steps S322 and S323 in the same manner as in steps S311 and S312.
If there is no mistake in the displayed text and there is a next text page in step S324 (Yes in step S324), in step S325, the user turns the page of the book and turns the turned page into the information processing apparatus SP. Have an image taken. Regarding the presence / absence of the next reading aloud page, the presence / absence of an adjacent page of the captured image may be detected, or whether or not a new page has been turned by the user may be detected.
Further, when the image of the book for two pages is included in the captured image, the automatic text conversion application cuts out the image of the next next page from the captured image, sends it to the server, and proceeds to step S316.
On the other hand, in step S324, if there is no next page to be converted into text (No in step S324), a notification of the end of text conversion is transmitted to the cloud CD.

In the cloud CD, the server stores the end position of the text in step S326 by the notification of the end of text conversion received from the information processing apparatus SP, and ends the text conversion process on the server side in step S327.

According to the information processing device SP having a text conversion function that operates in this way, when the information processing device SP with the mirror ML attached is set on the base BS and the book whose page is opened is imaged by the front camera, the imaged page is taken. Since the captured image of is displayed in the captured image display area 35 of the UI 31, the image of the book can be aligned while checking the captured image of the captured image display area 35, and the page of the book can be captured in the correct imaging range. Can be done. As a result, it is possible to eliminate page breaks and reduce mistakes when converting a book into text.
Further, according to the automatic text conversion application of the information processing device SP, the captured image of the book page is subjected to character recognition processing to extract the text, and the text is taken into the automatic text conversion application of the information processing device SP and displayed in the display area 32 of the UI 31. Since it is displayed, it is possible to digitize a paper-based book in a short working time.
Furthermore, according to the automatic text conversion application of the information processing device SP, an existing book is identified based on the extracted text and the data of the existing book is acquired. There is no unread, and the accuracy of text conversion of books can be improved.

In the operation examples of the translation function, the reading function, and the text conversion function, an example in which the mirror ML is set in the information processing apparatus SP provided with the fixed camera shown in FIG. 13 has been described, but these functions are shown in FIG. It can also be applied to the information processing apparatus 1 shown in FIG. 4 and the information processing apparatus 1 provided with the sliding rotary camera shown in FIGS. 20 and 21.
In the information processing apparatus 1 shown in FIGS. 1 to 4 and the information processing apparatus 1 shown in FIGS. 20 and 21, the information processing apparatus 1 is shown in FIG. 16 in a state where the information processing apparatus 1 is erected by sliding the display unit as necessary. Since the rotating camera can be set on the base BS and rotated forward from the rear side of the information processing device 1 to direct the optical axis of the rotating camera downward, the page to be read can be expanded without attaching the mirror ML. You can read the information processing. There is no need for a mirror mode that flips left and right. However, depending on the shooting settings, it may be necessary to flip it upside down.
Needless to say, the translation function, the reading aloud function, and the text conversion function can also be applied to the information processing device 1 provided with the rotating camera shown in FIG.

Next, another embodiment of the information processing apparatus SP shown in FIG. 13 will be described with reference to FIGS. 31 and 32. 31 and 32 are diagrams showing other embodiments of the information processing apparatus SP.
As shown in FIG. 31, in the information processing apparatus SP of this embodiment, the front camera FC is mounted on the upper end portion of the information processing apparatus main body SP1. The information processing apparatus main body SP1 has a predetermined thickness T1.
A mirror unit MU is attached to the information processing apparatus main body SP1 so as to cover the front camera FC. The mirror unit MU is provided with a groove MZ. Facing the figure, the right side of the groove MZ is the support portion MU1, and the left side is the reflection portion MU2.
The inside of the reflecting portion MU2 has an inclined wall surface and a wall surface covering the left and right ends thereof, and is opened downward and sideways on the information processing apparatus main body SP1 side.
A mirror ML is arranged on the inclined wall surface of the reflecting unit MU2 so as to be inclined in the optical axis direction of the front camera FC of the information processing apparatus main body SP1. A book placed with its pages opened on the surface on which the base BS is placed) can be imaged by the front camera FC by the mirror ML.
The groove MZ of the mirror unit MU is provided with a width wider than the thickness T1 of the information processing apparatus main body SP1. Therefore, when the mirror unit MU is attached to the information processing apparatus main body SP1, a gap is created. An attachment AT1 is arranged between the groove MZ of the information processing unit MU and the information processing apparatus main body SP1 so as to fill this gap.
The attachment AT1 is for closing the gap between the information processing unit MU and the groove MZ corresponding to the thickness T1 of the information processing device main body SP1 and fixing the information processing unit MU to the information processing device main body SP1.
As another embodiment, for example, as shown in FIG. 32, in the case of the information processing apparatus main body SP2 having a thickness T2 thicker than the thickness T1, an attachment AT2 thinner than the attachment AT1 shown in FIG. 31 is prepared.
By inserting the attachment AT2 into the groove MZ when the mirror unit MU is attached to the information processing device main body SP2, the mirror unit MU can be fixed to the information processing device main body SP2 without a gap in the groove MZ portion.

In the information processing apparatus SP configured in this way, by attaching attachments AT1 and AT2 for absorbing the thickness direction of the information processing apparatus main body SP1 and the information processing apparatus main body SP2 having different thicknesses, various thicknesses and cases are attached. Corresponding to the information processing apparatus SP, the mirror unit MU can be firmly fixed to the information processing apparatus main bodies SP1 and SP2.
That is, by preparing attachments AT1 and AT2 according to the thicknesses T1 and T2 of the information processing apparatus main bodies SP1 and SP2, the mirror unit MU can be attached to the information processing apparatus SP of various thicknesses to read a book. Become.

The present invention is not limited to the above-described embodiment, and modifications, improvements, and the like including omissions to the extent that the object of the present invention can be achieved are included in the present invention.
For example, matching with the content (text) of an existing book does not necessarily have to be done. Even if matching is performed, matching may be attempted by shooting the barcode of the book instead of shooting the title of the book.
In the above embodiment, an example of reading a book has been described, but the present invention is not limited to this, and the text conversion of a paper recording medium including a book such as a magazine is almost the same. Paper recording media are called paper leaves. In addition to paper leaves, the same is true for text conversion of things that are similar to paper displays, such as e-books.
In the above embodiment, the translation and reading aloud processes are performed, but these processes may be skipped and the text extracted from the captured image and the captured image may be displayed together on the UI 31.
In addition to displaying the text on the UI31, the format data (format) for inputting data such as receipts is registered in the memory, and the text including numbers is input in the column (input item) of the format. Then, for example, a receipt in which an amount, a date, an address, etc. are entered, or a payment slip for the receipt can be automatically generated.

For example, in the above-described embodiment, the connecting portion 12 is composed of two movable portions, a camera base portion 12-1 and a camera-oriented portion 12-2, but the connection portion 12 is not particularly limited to the above-described embodiment and is not limited to the main body portion. It suffices to have one or more movable parts having a volume smaller than 11.
In this case, it is sufficient that the camera 21 as the image pickup unit is arranged on a predetermined surface of one or more movable parts.
Further, a device having an arbitrary function can be mounted on an arbitrary number of movable portions (including a movable portion in which an imaging unit is arranged) of one or more movable portions.

For example, the information processing device 1 of the present invention may be used as an infrared remote controller (remote controller).
For example, according to one embodiment of the information processing apparatus 1 of the present invention, when a remote controller is photographed by a camera, the image is recognized and the function of each button is realized by the display and the touch panel of the information processing apparatus to process information. An infrared command may be issued from the device, or a command may be sent from the information processing device to the infrared transmitter by radio such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

For example, according to one embodiment of the information processing apparatus 1 of the present invention, it is possible to realize a function of navigating on an AR for an image taken by a 2-axis rotating camera, a 1-axis rotating camera, and a 360-degree camera. can.
In this case, the navigation system may be, for example, not displaying an image but only a voice.
Further, the information processing apparatus 1 may notify the corner by haptics, for example.
Further, in the information processing device 1, for example, the camera 21 may automatically rotate to notify the direction.

The translation, reading aloud, and text conversion functions shown in FIGS. 23 to 30 were realized by a system in which the information processing apparatus SP and the cloud CD were connected via a network, but the functions on the cloud CD side were realized by the information processing apparatus. Each function of translation, reading aloud, and converting into text may be realized by the information processing apparatus SP alone, including the SP.

As described above, the information processing apparatus 1 to which the present invention is applied can take various embodiments. In other words, the information processing apparatus 1 to which the present invention is applied may have the following configuration, and the embodiment thereof is not particularly limited.
That is, the information processing apparatus to which the present invention is applied (for example, the information processing apparatus 1 in FIG. 1) is
An imaging means for imaging (for example, the camera 21 in FIG. 2) and
A subject extraction means for extracting a subject from an captured image generated by the image pickup means, and a subject extraction means.
An information extraction means for extracting information indicated by the subject, and
Information analysis means for analyzing the information and
A generation means for generating an object based on the result of the analysis,
A presentation means for presenting the object together with the captured image,
To prepare for.

Further, the information processing apparatus to which the present invention is applied (for example, the information processing apparatus SP in FIG. 13) is
An image pickup means (for example, the front camera of the information processing apparatus SP in FIG. 13 (C)) for imaging a recording medium (such as a book in FIG. 13 (B)) on which a character string is arranged, and
A text data generation means (for example, an automatic translation reading application of the information processing apparatus SP of FIG. 13A) that generates text data corresponding to the character string based on the captured image captured by the imaging means.
A presentation means (for example, UI31 in FIG. 23) that presents both an image captured by the image pickup means and the text data.
To prepare for.
With this configuration, it is possible to shoot in a free posture while visually recognizing the camera viewpoint image. Further, according to this configuration, the user can browse the generated text data while checking the image of the captured image (position of the character string of the recording medium) and adjusting the recording medium, so that the text data can be generated without any mistake. can do.
The image pickup means (for example, the rotating camera 21 in FIG. 20) of the information processing apparatus is
An optical axis is rotatably supported at the end of the information processing apparatus main body (for example, the information processing apparatus 1 in FIG. 20) so as to face from the back side of the information processing apparatus main body to the front side where the text data is presented. NS.
With this configuration, text data can be generated without attaching a member such as a mirror that refracts the optical axis.
The image pickup means (for example, the front camera FC of FIG. 13C or FIG. 31) of the information processing apparatus SP is
An optical axis is provided at the end of the information processing apparatus main body SP1 so that the optical axis is directed in the normal direction of the display unit 13 on which the text data is presented.
The page of the recording medium (for example, the book of FIG. 13B) is configured to be imaged from the direction of the optical axis refracted through the mirror ML attached to the information processing apparatus main body SP1.
With this configuration, the information processing device SP can be placed upright on the base BS to image the pages of a book arranged on a flat surface, and the user has the information processing device main body SP1. You will be able to turn the pages of the book and adjust the position of the book.
The image pickup means (for example, the rotating camera 21 in FIG. 20) of the information processing apparatus 1 is
An optical axis is rotatably supported at the end of the information processing apparatus main body (for example, the main body portion 11 in FIG. 20) so as to face the front side on which the text data is presented from the back side of the information processing apparatus main body.
The first state (for example, in FIG. 20A, the first range (0 ° to 180 °)) in which the display unit 13 provided so as to be slidable on the front surface of the information processing device main body overlaps with the information processing device main body is set. As a range of motion
In the second state in which the display unit 13 is slid so that a part of the display unit 13 does not overlap with the information processing apparatus main body (for example, in FIG. 20B, the second range is wider than the first range). The range of motion (0 ° to 270 °) is defined as the range of motion. Further, the range of 270 ° or more is defined as the range of motion until the rotating camera 21 comes into contact with the end face of the slid display unit 13.
Note that 0 ° is a state in which the rotating camera 21 is housed in a groove or a notch in the main body 11 (a state in which the rotating camera 21 and the main body 11 overlap in the thickness direction), and 180 ° is a state in which the rotating camera 21 is overlapped in the thickness direction. Are in a linearly arranged state outside the main body 11. When it exceeds 180 °, the rotating camera 21 protrudes toward the front surface 11a of the main body portion 11, and in 270 °, the optical axis of the rotating camera 21 faces directly below the main body portion 11.
With this configuration, the information processing device 1 can be set up on the table BS and the pages of the book placed open on the surface on which the table BS is placed can be imaged without attaching the mirror ML. Therefore, the user can turn the pages of the book, adjust the position of the book, and the like without having to hold the information processing apparatus main body.
The information processing device 1 (for example, the information processing device 1 of FIG. 10 or FIG. 11) is
Information indicating the direction of the image pickup means is added to the image captured by the image pickup means.
With this configuration, the user can see what the situation of the information processing device 1 was at the time of taking a picture (the main body portion 11 was positioned horizontally and the rotating camera 21 was imaged with a vertically long angle of view. 11 is positioned vertically and the rotating camera 21 is photographed at a horizontally long angle of view, etc.).
The information processing device 1 (for example, the information processing device SP of FIG. 13 or FIG. 20) is
A correction means for correcting the captured image to an image suitable for generating the text data (for example, an automatic translation reading application of the information processing apparatus SP of FIG. 13 or FIG. 20).
To prepare for.
With such a configuration, misreading or unreading at the time of OCR can be reduced, and the character recognition accuracy of the text data can be improved.
The information processing apparatus SP (for example, the information processing apparatus SP of FIGS. 13 and 20 or a cloud CD) is
A processing means that performs at least one of translation and reading aloud of the text data (for example, an automatic translation reading aloud application of the information processing apparatus SP of FIGS. 13 and 20).
To prepare for.
With this configuration, the text data translated by the information processing apparatus SP can be viewed and read aloud.
The information processing apparatus SP (for example, the information processing apparatus SP of FIGS. 13 and 20 or a cloud CD) is
When the text data is incomplete as a sentence, a coupling means for combining with the text data obtained from another part of the captured image or the next captured image.
To prepare for.
With this configuration, the text data is combined and completed as one sentence, so that the user can browse and read aloud the sentence of the book without interruption in the information processing apparatus SP.

1 ... Information processing device 1-1 to 1-3 ... Information processing device 11 ... Main body 11a ... Front 11b ... Back 12 ... Connecting part 12-1 ... Camera base 12-2 ... Camera oriented part 13 ... Display part 14 ... Stand 21 ... Camera AD1, AD2 ... Advertisement AL ... Auxiliary line B1 ... Delete button B2 ... Under consideration Button B3 ... Holding button BS ... Stand CD ... Cloud CL ... Clip CO ... Object EP1, EP2 ... Converted price display G1, G2 ... Game image GC ... Game Characters GI1 to GI4 ... Guide information GS ... Gas station image HB ... Semi-transparent button MD ... User ML ... Mirror MN ... Restaurant menu MP ... Map MS ... Message OB ・ ・ ・ Original button OT ・ ・ ・ Original OP1, OP2 ・ ・ ・ Original price display OP ・ ・ ・ Past photos OW ・ ・ ・ Cooking name PL ・ ・ ・ People PQ ・ ・ ・ Problem PW ・ ・ ・ Place name QG ・・ ・ Problem image QP ・ ・ ・ Problem paper RC ・ ・ ・ Shutter remote control RM ・ ・ ・ Ramen shop line image SC ・ ・ ・ Screen SP ・ ・ ・ Information processing device with fixed camera SS ・ ・ ・ Sound source TB ・・ ・ Translation button TT ・ ・ ・ Translated text TW ・ ・ ・ Translated text U ・ ・ ・ User WK ・ ・ ・ Object Z1, Z2, Z3, Z4 ・ ・ ・ Rotating axis

Claims (19)

An imaging means that captures an image of a medium on which a character string is placed,
A separate mirror that refracts the optical axis direction of the imaging means, and
A transmission means for transmitting the captured image to the cloud via the mirror, and
A text data generation means that generates text data corresponding to the character string based on an image captured by the image pickup means, and a text data generation means.
An information processing device. For the captured image transmitted to the cloud, horizontal inversion, vertical inversion, rotation, curvature correction of the medium part in which the character string is arranged, trapezoidal correction of the medium part in which the character string is arranged, contrast adjustment, character string A correction means that corrects at least one of the cutout of only the medium part where the image is placed and the deletion of the image of the finger that has been reflected.
The information processing apparatus according to claim 1. A processing means that performs at least one of translation and reading aloud of the text data.
The information processing apparatus according to claim 1. From the text data of the medium on which the character string is arranged or the barcode information described in the medium, the reference data is presented by referring to the reference data predetermined to be associated with the past text data or the barcode information. Have a means of presentation,
The information processing apparatus according to claim 1. It has a presentation means for presenting both the captured image captured by the imaging means and the text data.
The information processing apparatus according to claim 1. When the text data is incomplete as a sentence, the text data is combined with the text data obtained from another part of the captured image or the next captured image to complete one sentence.
The information processing apparatus according to claim 1. An imaging means for imaging a recording medium on which a character string is arranged, and
A hinge portion that rotates the image pickup means so as to change the optical axis direction of the image pickup means, and a hinge portion.
A transmission means for transmitting the captured image to the cloud,
A text data generation means that generates text data corresponding to the character string based on an image captured by the image pickup means, and a text data generation means.
An information processing device. Horizontal inversion, vertical inversion, rotation, curvature correction of the medium part in which the character string is arranged, trapezoidal correction of the medium part in which the character string is arranged, contrast adjustment, and a medium in which the character string is arranged with respect to the captured image. A correction means that corrects at least one of the cropping of only the part and the deletion of the image of the finger that has been reflected.
7. The information processing apparatus according to claim 7. A processing means that performs at least one of translation and reading aloud of the text data.
7. The information processing apparatus according to claim 7. From the text data of the medium on which the character string is arranged or the barcode information described in the medium, the text data in the past or the reference data predetermined to be associated with the barcode information is referred to, and the reference data is presented. Have a means of presentation,
The information processing apparatus according to claim 7. It has a presentation means for presenting both the captured image captured by the imaging means and the text data.
The information processing apparatus according to claim 7. When the text data is incomplete as a sentence, the text data is combined with the text data obtained from another part of the captured image or the next captured image to complete one sentence.
The information processing apparatus according to claim 7. An image pickup unit that captures the subject and
A display unit that overlaps with the image pickup unit in the thickness direction and displays an image captured image including the subject imaged by the image pickup unit.
The image pickup unit is supported by a first rotation axis parallel to the lateral direction of the display unit so as to change the angle formed by the optical axis direction of the image pickup unit and the normal direction with respect to the display unit. However, the hinge part that supports the main body part so as to be relatively rotatable,
Equipped with
The image pickup unit is
A smart phone capable of relatively parallel movement with respect to the display unit from a position where the first rotation axis overlaps with the display unit in the thickness direction to a position where the first rotation axis does not overlap with the display unit. The image pickup unit is
A drive unit that drives the hinge unit and rotates the image pickup unit,
A control unit that controls the drive unit so as to rotate the image pickup unit in an instruction direction in response to a movement instruction to the image pickup unit.
including,
The smart phone according to claim 13. The image pickup unit is
A hinge portion that is rotatably supported by a second rotation axis parallel to the longitudinal direction of the display portion,
including,
The smart phone according to claim 13. The image pickup unit is
A hinge portion that is rotatably supported by the second rotating shaft at a position that does not overlap with the display portion in the thickness direction.
including,
The smart phone according to claim 13. The display unit
A touch panel for performing the translation instruction by touch operation,
including,
The smart phone according to claim 13. The hinge part
A first rotation axis arranged parallel to the lateral direction of the display unit,
A second rotation axis arranged parallel to the longitudinal direction of the display unit,
Equipped with
When the image pickup unit is rotated around the second rotation axis, the first rotation axis does not rotate in accordance with the rotation of the image pickup unit.
The smart phone according to any one of claims 13 to 17. A main body unit having a larger volume than the image pickup unit, in which a display unit for displaying an image captured image including the subject imaged by the image pickup unit is arranged on a predetermined surface.
The imaging unit is supported by a rotation axis parallel to the lateral direction of the display unit so as to change the angle formed by the optical axis direction of the imaging unit and the normal direction of the display unit. A hinge part that supports the body part so that it can rotate relatively, and
A recess provided on the other surface of the main body portion facing the one surface and capable of accommodating the image pickup portion rotated to the other one surface side by the hinge portion so as to overlap the main body portion in the thickness direction.
The first sensor, which is arranged in the main body and detects the direction of the main body by itself,
A second sensor, which is arranged in the image pickup unit and detects the direction of the image pickup unit by itself,
Equipped with
Based on the direction of the image pickup unit detected by the second sensor, the direction information is given to the digital photo file.
smart phone.

Images