Avatar of XGIS
XGISFlag for Australia asked on

What is the best online system to use to subset large spatial datasets for extraction of images?

I have multiple individual large spatial datasets, grids/rasters around 80Gb.  What is the best way to deal with delivery of this information online.  The proposed method is to allow the user to select an area eg rectangle on a google map interface. The rectangle geographic extent coordinates are used to issue the subset instructions and generate an output image of the area of interest. Instead of processing the image extraction from the large dataset I was looking at a solution that stitches image tiles together at resolution based on the area of interest. My questions relate to the following areas;

1. Type of OS / Scripting Language or Framework of the server (eg Linux/Python Windows/.NET (my primary technology)). Pls note CLOUD systems are NOT an option.
2. Perform Subsetting on a remote server or Supply Subset PNG8 images on the main server that get stitched together.
3. Open source systems that already achieve this.

Geoserver appears to be a potential solution.
GIS/GPS ProgrammingWeb DevelopmentWeb Servers

Avatar of undefined
Last Comment
XGIS

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
Scott Fell

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
SOLUTION
tbsgadi

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
ASKER
XGIS

Thankyou Padas and tbsgadi, both very useful answers.  
My scenario has changed slightly after doing further calculations that show I would require at least 400Gb to store my shapefiles for my 5m contour layer and 1.5Tb to store images at 150dpi. Since I dont have googles budget I have opted to store the 100 part processed grid tiles on the server (70gb) which will form the basis of any geoprocessing using GDAL and Python. Generated  thematic data layers and images will then be made available for client delivery as compressed downloads.  As for the web server I am looking at Telerik silverlight RadMap (which I already have licenced) which has a lot of basic resources for doing this kind of task, eg various providers, projections etc.

Any further comments are welcome as I am still in the planning stage. Just to clarify I do not plan to serve the Grid tiles through the web server.  The WMS will only be used to locate an area and draw a shape around it which will then be stored and used for any subsequent processing.
Scott Fell

It sounds like you have everything under control
ASKER
XGIS

Both answers were equally useful in helping me reach a decision to this point in time. Thankyou for your time.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck