• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3158
  • Last Modified:

Match, transform, and drape 2D images to 3D mesh

I am starting a new project, but find I am out of my league and need some insight from the gurus.  I am happy to learn what needs to be done on my own, but time is scarce and rather than spend days working on something that may not work, I would appreciate if the community would mind pointing me in the right direction (i.e., best practices, relevant papers, or even just your thoughts).  

Situation:  I have an m x 2 array of images (relative order is known) and high-resolution (1mm x 1mm) elevation data for an area covering L x W meters.  Objects in the images have fine detail, are heterogeneous, and for all intents and purposes, randomly distributed.

Objective:  
1.  Create a mesh from elevation data
2.  Find homographies between mesh and image (SIFT in 3D?)
3.  Position camera in space to display desired position and scale
4.  Transform images as needed and drape over mesh
5.  Blend images (multiresolution spline?) to completely eliminate seams
6.  Apply bump map (presumably from original photos?) to give a more natural appearance
7.  Create red/cyan anaglyph on current view and export as TIF (will be large, but dims > supported by JPEG)

I am most comfortable programming in VB .NET and C#, but will take a crash course in C++ if I need to (I probably should anyway).

Thanks for any help!
0
pigouvian
Asked:
pigouvian
2 Solutions
 
ozoCommented:
Are the m x 2 images m stereo pairs?
http://vision.stanford.edu/~birch/p2p/alg/
0
 
pigouvianAuthor Commented:
Ozo -- thanks for the quick response!  Unfortunately, the images are not stereo pairs.  It just so happened that it only took two images to capture the width of the area under study with ~20% overlap.
0
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
ozoCommented:
If you already have the height data from some other source then you don't need to derive it from the image alignment.
steriopsis algorithms may still be useful for matching features in the overlap area.
or maybe you can derive the alignment from the height data instead of vice versa if you know both camera positions to the same accuracy as the resolution of the images.
0
 
ozoCommented:
So is your question about projection mapping?
http://www3.interscience.wiley.com/cgi-bin/abstract/77502997/ABSTRACT
0
 
pigouvianAuthor Commented:
From a cursory look into the concepts of stereopsis and projection mapping, it seems like we are on the right track.

W.r.t. projection mapping, I would agree this seems to be an integral method to the construction of the image.  However, there are two additional requirements I must find a solution for.  First, there are lots of images and lots of runs.  As such, I must find a way to automate the process of finding (at least I assume I need to find) point-wise correspondence between features on the generated mesh with features of the image* -- something analogous to what SIFT and other matching algorithms do for finding homographies between two 2D images to create panoramas.  Second, since there is more than one image being projected onto the surface, I need to be able blend the overlap in the images so they are seamless.  My original thought was I could create the panorama and then drape over the mesh, but I really only want to deform the images once.  

W.r.t stereopsis algorithms, this seems the ticket to complete the last step.  Thanks!


* note: each image is named such that the relative position is known, e.g., file001x_10000y_00600.jpg. However, I do not know the exact width of the area photographed, the coordinates of the optical center of the camera, or even which pixel the coordinate in the filename corresponds to.  It seems the only thing I can do with this data is to order the images.
0
 
Jose ParrotGraphics ExpertCommented:
Hi, lets see if I can join ozo in helping you.

Some things are unclear for me, so please clarify them. After each point I post my understanding, please tell us if they are right or wrong or partial....

1.  Create a mesh from elevation data
You may create a 3D mesh from the elevation data simply defining a 2D array, and load the elevation data into each cell. For example: if L x W = 5 x 10 meters and you have data for each millimeter, then your elevation data has  50,000,000 values and your array (as the mesh as well) is 5000 x 10000.

2.  Find homographies between mesh and image (SIFT in 3D?)
You want find how to trim the images and glue the pieces together such that you have the complete area coincident to the elevation map. As there are superpositions, you can determine the common area in 2 neighbour images and cut them by the middle of the common area. As you know the coordinates of each image, then you know where they are in the mesh.

3.  Position camera in space to display desired position and scale
By scale I understand how you determine the height of each cell, taking in account the value in the elevation data. Position a camera is a bit more complicated, because you'll need a complete 3D SDK. I don't want to discourage you, but just a single camera has the following attributes: Position (x,y,z), View point (x,y,z), Tilt (angle), Aperture Lens (angle). And rendering what appear in the Viewport isn't a simple task. My suggestion is to use OpenGL or DirectX. I strongly recommend OpenGL. Not so easy too, but it will short your development time a lot! By the other hand, you can select a small piece of your array and export it to Excel and make a 3D graph (very simple!).

4.  Transform images as needed and drape over mesh
I can't figure out which transforms are you looking for.

5.  Blend images (multiresolution spline?) to completely eliminate seams
The images are supposed to be in the same resolution, right? If so, what is the purpose of multiresolution splines? Ah! If you are looking for spline, I don't think  you need them, as you have a mesh with 1 millimeter resolution.

6.  Apply bump map (presumably from original photos?) to give a more natural appearance
To use photos from the terrain (?) you must resize them such each pixel corresponds to each elevation cell. You may want use IrfanView for this function. As soon you have the photos in the right resolution, you call use OpenGL for bump mapping.

7.  Create red/cyan anaglyph on current view and export as TIF (will be large, but dims > supported by JPEG)
You can also use OpenGL to create two images, each one equivalent to right/left eye, and filter them with the colors. To join the images into a single one, you need Photoshop, Corel Photopaint or cheaper (good!) PaintShop.

About C++, it is prefferable over VB or C#, although these are viable languages. Please note: C# has a Loooooooow performance, really bad. If you plan to use OpenGL or other 3D engine, most are made to integrate easly to C++. Take a look at nehe.gamedev.net for OpenGL lessons.

Finally, without discourage you... That is a LOT of work!!! Let me suggest develop the image and mesh generation program and adpt a graphics 3D tool to do the 3D presentation, like TrueSpace (version 3 is free) or POV-RAY, also free.

As you see.. this is not ONE question... it is a set of them...

Jose
0
 
Computer101Commented:
Forced accept.

Computer101
EE Admin
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Microsoft Windows 7 Basic

This introductory course to Windows 7 environment will teach you about working with the Windows operating system. You will learn about basic functions including start menu; the desktop; managing files, folders, and libraries.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now