Solved

# Match, transform, and drape 2D images to 3D mesh

Posted on 2007-10-04
3,088 Views
I am starting a new project, but find I am out of my league and need some insight from the gurus.  I am happy to learn what needs to be done on my own, but time is scarce and rather than spend days working on something that may not work, I would appreciate if the community would mind pointing me in the right direction (i.e., best practices, relevant papers, or even just your thoughts).

Situation:  I have an m x 2 array of images (relative order is known) and high-resolution (1mm x 1mm) elevation data for an area covering L x W meters.  Objects in the images have fine detail, are heterogeneous, and for all intents and purposes, randomly distributed.

Objective:
1.  Create a mesh from elevation data
2.  Find homographies between mesh and image (SIFT in 3D?)
3.  Position camera in space to display desired position and scale
4.  Transform images as needed and drape over mesh
5.  Blend images (multiresolution spline?) to completely eliminate seams
6.  Apply bump map (presumably from original photos?) to give a more natural appearance
7.  Create red/cyan anaglyph on current view and export as TIF (will be large, but dims > supported by JPEG)

I am most comfortable programming in VB .NET and C#, but will take a crash course in C++ if I need to (I probably should anyway).

Thanks for any help!
0
Question by:pigouvian

LVL 84

Expert Comment

Are the m x 2 images m stereo pairs?
http://vision.stanford.edu/~birch/p2p/alg/
0

LVL 84

Expert Comment

0

Author Comment

Ozo -- thanks for the quick response!  Unfortunately, the images are not stereo pairs.  It just so happened that it only took two images to capture the width of the area under study with ~20% overlap.
0

LVL 84

Accepted Solution

If you already have the height data from some other source then you don't need to derive it from the image alignment.
steriopsis algorithms may still be useful for matching features in the overlap area.
or maybe you can derive the alignment from the height data instead of vice versa if you know both camera positions to the same accuracy as the resolution of the images.
0

LVL 84

Expert Comment

http://www3.interscience.wiley.com/cgi-bin/abstract/77502997/ABSTRACT
0

Author Comment

From a cursory look into the concepts of stereopsis and projection mapping, it seems like we are on the right track.

W.r.t. projection mapping, I would agree this seems to be an integral method to the construction of the image.  However, there are two additional requirements I must find a solution for.  First, there are lots of images and lots of runs.  As such, I must find a way to automate the process of finding (at least I assume I need to find) point-wise correspondence between features on the generated mesh with features of the image* -- something analogous to what SIFT and other matching algorithms do for finding homographies between two 2D images to create panoramas.  Second, since there is more than one image being projected onto the surface, I need to be able blend the overlap in the images so they are seamless.  My original thought was I could create the panorama and then drape over the mesh, but I really only want to deform the images once.

W.r.t stereopsis algorithms, this seems the ticket to complete the last step.  Thanks!

* note: each image is named such that the relative position is known, e.g., file001x_10000y_00600.jpg. However, I do not know the exact width of the area photographed, the coordinates of the optical center of the camera, or even which pixel the coordinate in the filename corresponds to.  It seems the only thing I can do with this data is to order the images.
0

LVL 18

Assisted Solution

Hi, lets see if I can join ozo in helping you.

Some things are unclear for me, so please clarify them. After each point I post my understanding, please tell us if they are right or wrong or partial....

1.  Create a mesh from elevation data
You may create a 3D mesh from the elevation data simply defining a 2D array, and load the elevation data into each cell. For example: if L x W = 5 x 10 meters and you have data for each millimeter, then your elevation data has  50,000,000 values and your array (as the mesh as well) is 5000 x 10000.

2.  Find homographies between mesh and image (SIFT in 3D?)
You want find how to trim the images and glue the pieces together such that you have the complete area coincident to the elevation map. As there are superpositions, you can determine the common area in 2 neighbour images and cut them by the middle of the common area. As you know the coordinates of each image, then you know where they are in the mesh.

3.  Position camera in space to display desired position and scale
By scale I understand how you determine the height of each cell, taking in account the value in the elevation data. Position a camera is a bit more complicated, because you'll need a complete 3D SDK. I don't want to discourage you, but just a single camera has the following attributes: Position (x,y,z), View point (x,y,z), Tilt (angle), Aperture Lens (angle). And rendering what appear in the Viewport isn't a simple task. My suggestion is to use OpenGL or DirectX. I strongly recommend OpenGL. Not so easy too, but it will short your development time a lot! By the other hand, you can select a small piece of your array and export it to Excel and make a 3D graph (very simple!).

4.  Transform images as needed and drape over mesh
I can't figure out which transforms are you looking for.

5.  Blend images (multiresolution spline?) to completely eliminate seams
The images are supposed to be in the same resolution, right? If so, what is the purpose of multiresolution splines? Ah! If you are looking for spline, I don't think  you need them, as you have a mesh with 1 millimeter resolution.

6.  Apply bump map (presumably from original photos?) to give a more natural appearance
To use photos from the terrain (?) you must resize them such each pixel corresponds to each elevation cell. You may want use IrfanView for this function. As soon you have the photos in the right resolution, you call use OpenGL for bump mapping.

7.  Create red/cyan anaglyph on current view and export as TIF (will be large, but dims > supported by JPEG)
You can also use OpenGL to create two images, each one equivalent to right/left eye, and filter them with the colors. To join the images into a single one, you need Photoshop, Corel Photopaint or cheaper (good!) PaintShop.

About C++, it is prefferable over VB or C#, although these are viable languages. Please note: C# has a Loooooooow performance, really bad. If you plan to use OpenGL or other 3D engine, most are made to integrate easly to C++. Take a look at nehe.gamedev.net for OpenGL lessons.

Finally, without discourage you... That is a LOT of work!!! Let me suggest develop the image and mesh generation program and adpt a graphics 3D tool to do the 3D presentation, like TrueSpace (version 3 is free) or POV-RAY, also free.

As you see.. this is not ONE question... it is a set of them...

Jose
0

LVL 1

Expert Comment

Forced accept.

Computer101
0

## Featured Post

Most everyone who has done any programming in VB6 knows that you can do something in code like Debug.Print MyVar and that when the program runs from the IDE, the value of MyVar will be displayed in the Immediate Window. Less well known is Debug.Asse…
Enums (shorthand for ‘enumerations’) are not often used by programmers but they can be quite valuable when they are.  What are they? An Enum is just a type of variable like a string or an Integer, but in this case one that you create that contains…
Get people started with the process of using Access VBA to control Outlook using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Microsoft Outlook. Using automation, an Access applic…
Show developers how to use a criteria form to limit the data that appears on an Access report. It is a common requirement that users can specify the criteria for a report at runtime. The easiest way to accomplish this is using a criteria form that a…