?
Solved

I have 2 gigs of data that needs to be accessed often and quickly ...

Posted on 2006-06-01
8
Medium Priority
?
260 Views
Last Modified: 2010-04-01

So, I'll have about 2 gigs of terrain data.

I'll be needing to access 20k to multi-megabyte chunks of it during real-time rendering of the terrain.

I have several books regarding game development, but I've not come upon this topic yet.

Any guesses on the best way to do this?

Any good books you know of?


P.S. I posted this in the C++ section because there may be something built in to C++ that I can use... the Game section might've been okay but I'll try that next if I don't get anything here
0
Comment
Question by:oxygen_728
  • 3
  • 2
  • 2
  • +1
8 Comments
 
LVL 30

Accepted Solution

by:
Axter earned 1000 total points
ID: 16805882
You can open the file, and seek to the location of the file you which to read.

For speed, avoid using the iostream C++ method, and instead use C file functions (fopen, fread).
0
 

Author Comment

by:oxygen_728
ID: 16805890
Thanks Axter, I'm going to leave this question open til later on... just to see if any other experts have some input

Much appreciated
0
 
LVL 2

Assisted Solution

by:seet82
seet82 earned 600 total points
ID: 16806023
i believe the format of the file is defined by other program and you can't change the format?
cause the format of the file can determine how you access the file and might allow you to use random access to get the information you want if the file is so called well formated
say, you are rendering a certain portion of the 3d terrain at certain coordinates which supposed is in tile 20
then you know that information in tile 20 is located at certain position in the file, you can then use random access to quickly access the information
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 4

Assisted Solution

by:e_tadeu
e_tadeu earned 400 total points
ID: 16806388
If you have plenty of RAM you can speed up a lot doing a memory map on the file!

http://www.informit.com/guides/content.asp?g=cplusplus&seqNum=213
0
 
LVL 4

Assisted Solution

by:e_tadeu
e_tadeu earned 400 total points
ID: 16806865
What is the format of the input file?

Another option would be to use the HDF5 library (but your file need to be in HDF5 format):

http://hdf.ncsa.uiuc.edu/HDF5/
0
 

Author Comment

by:oxygen_728
ID: 16811463
seet82: Ya, I can format the data however I want.. .I"m actually converting 20 gigs of SRTM elevation data into 2 gigs of usable data. Random Access may be nice, or perhaps i'll have a table of seek positions.

E_tadeu: I'll check out the memory map

e_tadeu: I can format the file however I want... i'm tackling this early =)

0
 
LVL 2

Assisted Solution

by:seet82
seet82 earned 600 total points
ID: 16813192
you might wanna look at chunk based file format so that a table of seek positions is not needed
a chunk based files formats is roughly like this

size of next chunk
data of current chunk
size of next chunk
data of current chunk
size of next chunk
data of current chunk

thus to find data, you will have to read the first few bytes(number of bytes is determine by how big each chunk will be) to determine the size of the 1st chunks
then you might wanna read the first few bytes of chunk 1 and see if it contains the info you want
if it does not, you will skip the file to the next chunk (based on the size of the current chunk that you have read earlier on)

the process is repeated until you get the chunk you want

i would say that the chunk based file format works a bit like linked list
0
 

Author Comment

by:oxygen_728
ID: 16814760
Thanks seet82.. and thanks everybody for your input
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Templates For Beginners Or How To Encourage The Compiler To Work For You Introduction This tutorial is targeted at the reader who is, perhaps, familiar with the basics of C++ but would prefer a little slower introduction to the more ad…
Article by: evilrix
Looking for a way to avoid searching through large data sets for data that doesn't exist? A Bloom Filter might be what you need. This data structure is a probabilistic filter that allows you to avoid unnecessary searches when you know the data defin…
The viewer will learn how to use the return statement in functions in C++. The video will also teach the user how to pass data to a function and have the function return data back for further processing.
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.
Suggested Courses

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question