• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 263
  • Last Modified:

I have 2 gigs of data that needs to be accessed often and quickly ...


So, I'll have about 2 gigs of terrain data.

I'll be needing to access 20k to multi-megabyte chunks of it during real-time rendering of the terrain.

I have several books regarding game development, but I've not come upon this topic yet.

Any guesses on the best way to do this?

Any good books you know of?


P.S. I posted this in the C++ section because there may be something built in to C++ that I can use... the Game section might've been okay but I'll try that next if I don't get anything here
0
oxygen_728
Asked:
oxygen_728
  • 3
  • 2
  • 2
  • +1
5 Solutions
 
AxterCommented:
You can open the file, and seek to the location of the file you which to read.

For speed, avoid using the iostream C++ method, and instead use C file functions (fopen, fread).
0
 
oxygen_728Author Commented:
Thanks Axter, I'm going to leave this question open til later on... just to see if any other experts have some input

Much appreciated
0
 
seet82Commented:
i believe the format of the file is defined by other program and you can't change the format?
cause the format of the file can determine how you access the file and might allow you to use random access to get the information you want if the file is so called well formated
say, you are rendering a certain portion of the 3d terrain at certain coordinates which supposed is in tile 20
then you know that information in tile 20 is located at certain position in the file, you can then use random access to quickly access the information
0
Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

 
e_tadeuCommented:
If you have plenty of RAM you can speed up a lot doing a memory map on the file!

http://www.informit.com/guides/content.asp?g=cplusplus&seqNum=213
0
 
e_tadeuCommented:
What is the format of the input file?

Another option would be to use the HDF5 library (but your file need to be in HDF5 format):

http://hdf.ncsa.uiuc.edu/HDF5/
0
 
oxygen_728Author Commented:
seet82: Ya, I can format the data however I want.. .I"m actually converting 20 gigs of SRTM elevation data into 2 gigs of usable data. Random Access may be nice, or perhaps i'll have a table of seek positions.

E_tadeu: I'll check out the memory map

e_tadeu: I can format the file however I want... i'm tackling this early =)

0
 
seet82Commented:
you might wanna look at chunk based file format so that a table of seek positions is not needed
a chunk based files formats is roughly like this

size of next chunk
data of current chunk
size of next chunk
data of current chunk
size of next chunk
data of current chunk

thus to find data, you will have to read the first few bytes(number of bytes is determine by how big each chunk will be) to determine the size of the 1st chunks
then you might wanna read the first few bytes of chunk 1 and see if it contains the info you want
if it does not, you will skip the file to the next chunk (based on the size of the current chunk that you have read earlier on)

the process is repeated until you get the chunk you want

i would say that the chunk based file format works a bit like linked list
0
 
oxygen_728Author Commented:
Thanks seet82.. and thanks everybody for your input
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

  • 3
  • 2
  • 2
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now