Apple patent is for generating 3D model using portable electronic device recording

Apple wants you to be able to use your iPhone, iPod touch and perhaps your iPad to be able to navigate a real-world environment via 3D images A new company patent (number 20100188503) has appeared at the US Patent & Trademark Office for generating a three-dimensional model using a portable electronic device recording.

According to Apple, an electronic device can load and provide an initial display of a three dimensional model (e.g., of an environment or of an object). As the user moves the electronic device, motion sensing components, positioning circuitry, and other components can detect the device movement and adjust the displayed portion of the three-dimensional model to reflect the movement of the device.

By walking with the device in the user’s real environment, a user can virtually navigate a representation of a three-dimensional environment. In some embodiments, a user can record an object or environment using an electronic device, and tag the recorded images or video with movement information describing the movement of the device during the recording. The recorded information can then be processed with the movement information to generate a three-dimensional model of the recorded environment or object. The inventors are Richard Tsai, Andrew Just and Brandon Harris.

This is directed to systems and methods for navigating a scene or around an object in three dimensions using deterministic movement of an electronic device. This is also directed to systems and methods for generating a three-dimensional model of a scene or of an object for subsequent three-dimensional navigation by recording the scene using an electronic device. The inventors are Richard Tsai, Andrew Just and Brandon Harris.

Here’s Apple’s background and summary of the invention: “Users of electronic devices can view various information displayed by the devices. For example, a user can direct an electronic device to display a representation of a three-dimensional object (e.g., a photograph), or allow a user to navigate a representation of a location or of a virtual world (e.g., navigate through a series of images representing a city, such as a series of images in a street view of a mapping application). A user can direct the device to display such information using any suitable approach, including for example by selecting the information from a locally stored or remotely accessed source using an input mechanism. Once selected, the information can be displayed.

“Because a planar electronic device display can inherently be only a two-dimensional or planar display, the device can, at any given moment, provide only a partial representation of the displayed information. For example, when a three dimensional object is displayed, the electronic device can only display the object from a single perspective. To view other perspectives, the user may be required to select a different image representing the same object, or provide an input (e.g., selecting a button or dragging a mouse) to cause the object to rotate or spin, providing other perspectives for the object.

“Similarly, to navigate a representation of a three-dimensional environment, the electronic device can require the user to provide successive inputs using the input mechanism to move through the environment and change the displayed images to reflect the user’s movement through the environment. In some cases, however, a user may not be able to provide an input to an input mechanism (e.g., the user’s hands are busy). Another mechanism may then be necessary to allow a user to view other portions of a displayed three-dimensional object or three-dimensional environment.

“In addition, users typically may not have an easy mechanism for generating three-dimensional representations of three-dimensional objects. Typically, such representations can require taking several images of an object from fixed and known positions and orientations relative the object, and subsequently processing the images to allow users to navigate between images to view all perspectives of the three-dimensional object or event. In particular, the user may be required to provide information related to the relative position and orientation of the lens for each image to allow for subsequent navigation of the images. This can be especially difficult without specialized equipment and prevent most users from generating three-dimensional models of objects or environments.

“This is directed to systems and methods for navigating three-dimensional environments and viewing three-dimensional objects on an electronic device display based on deterministic movement of the electronic device. This is also directed to systems and methods for recording a video of a three-dimensional environment or three-dimensional object, and processing the video to generate a three-dimensional model that can be navigated in an order other than that of the recording.

“In some embodiments, a user can direct an electronic device to display information that can be associated with three-dimensional navigation such as, for example, three-dimensional models of environments or objects. For example, a user can direct a user to access a mapping application that provides images of what can be seen in some or any direction from a particular location. As another example, a user can direct an electronic device to display a video game in which a user may navigate a virtual world and see, in any direction, what the virtual world resembles (e.g., rendered images of the virtual world’s appearance from any location in the virtual world). As still another example, a user can direct the electronic device to display a three-dimensional object (e.g., an object for sale) that the user can manipulate or view from different angles

“To change the particular display of the environment or object without requiring an input from a dedicated input mechanism, the electronic device can include a motion sensing component (e.g., an accelerometer) operative to detect movements of the electronic device. When the device displays information associated with three-dimensional navigation, the electronic device can monitor the motion-sensing component for device movement and change the displayed information to reflect the output of the motion sensing component.

“The change in appearance of the three-dimensional environment or object displayed by the device can be correlated to the device movement using any suitable approach, including for example using a linear approximation (e.g., as the user tilts the device, the electronic device displays the environment or object as if the user’s perspective of the object was tilted in the same manner as the device). In effect, the electronic device display can provide a window, movable in three dimensions, into the three-dimensional environment, or for viewing the three-dimensional object.

“In some embodiments, the user can direct the electronic device to record a three-dimensional environment or object to generate an interactive three-dimensional model. Using a lens of the electronic device, the electronic device can record an environment or object as a user moves the device. The recording can be simultaneously tagged with the output of the motion-sensing component, positioning information, or both to define the spatial relationship between video frames of the recording. The video can then be processed (e.g., on a host device) to generate a three-dimensional model using the images and spatial relationships. Any other electronic device can then load the generated model and allow the user of the other electronic device to navigate the three-dimensional model (e.g., using deterministic movements).”