Write database table to a stream and read stream back into other equivalent database table

Hi Experts,
I would like to design a fast algorithm that can take items in a database table and spit them out to a stream.  I'm new with stream programming - but I know there is a way to write to a file as a stream or treat sockets as the stream.  Can someone point out how to set this up?   And how would I delimit my records?

Many thanks,
Mike
LVL 1
threadyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jkrCommented:
Well, wouldn't it be easier to dump the table to a file and then reload it instead of doing that by hand? The problem with your approach is that you pretty much have to customise that for every table, since the filed definitions will vary...

BTW, which DBMS and library are you using?
0
threadyAuthor Commented:
Hi jkr, it's more complicated than that - I'm asking a more basic question than what I need - I am using Postgres.  Basically, I will use this with streams and to do synchronization between databases (sort of).  I have database objects smart enough to know how to save their own data, etc....  I've started by using XML - I'm hoping it's not too slow with all the text and extra lengths because of tag identifiers, etc.  Do you know of a library that can zip as I write to a file?

Mike
0
evilrixSenior Software Engineer (Avast)Commented:
>> Basically, I will use this with streams and to do synchronization between databases (sort of).
Why don't you just use database replication?
http://www.postgresql.org/about/news.233

0
Cloud Class® Course: Ruby Fundamentals

This course will introduce you to Ruby, as well as teach you about classes, methods, variables, data structures, loops, enumerable methods, and finishing touches.

threadyAuthor Commented:
I can't replicate the database tables - some tables will need to have minor differences (private data not to be shared with others)....
0
evilrixSenior Software Engineer (Avast)Commented:
BTW: Since you are using Postgres, why did you post to the MySQL Zone? There is a Postgres zone, I'm sure you would have been better off posting to there (I know MySQL but not Postgres!).
http://www.experts-exchange.com/Database/PostgreSQL/
0
evilrixSenior Software Engineer (Avast)Commented:
>> I can't replicate the database tables - some tables will need to have minor differences (private data not to be shared with others)....
Trying to synchronize manually it not going to be a trivial task. Writing the code to read from one and write to another is pretty simply. The problem is going to be managing concurrency. Have you given any thought to how you are going to do this?

0
evilrixSenior Software Engineer (Avast)Commented:
BTW: You might want to consider using ADO.Net for this.
http://en.wikipedia.org/wiki/ADO.NET
0
threadyAuthor Commented:
evilrix - it's not full synchronization... It's a simplification - and it took me about a year to figure out how to do it (to answer your question about whether or not I thought about how I was going to do this)... . I've also got 3 years of database objects coded that use ODBC... can't switch to nicer newer APIs this late in the game unfortunately.... :-)   Thanks for your ideas & suggestions.

Mike
0
evilrixSenior Software Engineer (Avast)Commented:
To be honest I'm a little at a lose as to what you actual question is then? You just want to know how to open and write a file stream in C++? See below.

http://www.cplusplus.com/doc/tutorial/files.html
http://www.cplusplus.com/reference/iostream/
#include <fstream>
 
int main()
{
	std::fstream fs("c:\\temp\\test.txt", std::ios::out | std::ios::in | std::ios::app);
	fs << "Hello World" << std::endl;
}

Open in new window

0
threadyAuthor Commented:
Hi evilrix:

I want to know how to link up functions - so I can pipe a stream one way or the other (kind of like OpenSSL BIO objects in case you're familiar with them)..   It may be a silly question, but I've just never piped one function's input stream to another.... I just need some background and maybe if you know about a compression "stream" that I can write to as well....

Mike
0
evilrixSenior Software Engineer (Avast)Commented:
>> so I can pipe a stream one way or the other
C++ IO streams aren't really designed to be piped to each other. You'll have to read from one stream and write to another. A stream is really just a fancy name for a common interface that wraps various data buffers (whether it's file or memory or other) and presents it in a uniform and extensible way.

>> and maybe if you know about a compression "stream" that I can write to as well....
If you are serious about this you should take a look at boost iostreams. They allow you to write your own streams based upon the standard ones (they expose the same interface so they can be used as drop in replacements) but with the added advantage that you can implement filers and attach them to the stream (so a filer could zip as you write and unzip as you read). There's a fair bit of reading to do but since this project has been 3 years in the making a few more hours RTM won't hurt :)

http://www.boost.org/libs/iostreams/doc/
http://www.boost.org/libs/iostreams/doc/classes/bzip2.html
http://www.boost.org/libs/iostreams/doc/classes/gzip.html
http://www.boost.org/libs/iostreams/doc/classes/zlib.html

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
threadyAuthor Commented:
exactly what I was looking for!  Thank you very much!  Many thanks!  :-)  (I think that's the one that a colleague had used a few years back)...
0
evilrixSenior Software Engineer (Avast)Commented:
One last thing. Building Boost (on Windows) can be a painful experience since it uses its own proprietary build system (BJam). Fortunately, you can download precompiled binaries for Windows from the following site: http://www.boost-consulting.com/products/free

I hope this helps and good luck.

-Rx.
0
threadyAuthor Commented:
Thanks for the luck - I've got a serious load of work to do and only a few months to get it all done & tested....  It's good not to feel so alone doing it....  And thanks again.
0
threadyAuthor Commented:
Oh and best to you as well.
0
evilrixSenior Software Engineer (Avast)Commented:
>> It's good not to feel so alone doing it...
Always here to help -- well, unless my wife is giving me grief for being at the PC "again" :)
Seriously though, if you get stuck just shout.

>> Oh and best to you as well.
And the best to you too my friend.

-Rx.
0
threadyAuthor Commented:
>> Always here to help -- well, unless my wife is giving me grief for being at the PC "again" :)
Haha - Don't I hear ya....  :-)  
Cheers,
Mike
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C++

From novice to tech pro — start learning today.