We help IT Professionals succeed at work.

Write database table to a stream and read stream back into other equivalent database table

thready
thready asked
on
Hi Experts,
I would like to design a fast algorithm that can take items in a database table and spit them out to a stream.  I'm new with stream programming - but I know there is a way to write to a file as a stream or treat sockets as the stream.  Can someone point out how to set this up?   And how would I delimit my records?

Many thanks,
Mike
Comment
Watch Question

jkr
BRONZE EXPERT
Top Expert 2012

Commented:
Well, wouldn't it be easier to dump the table to a file and then reload it instead of doing that by hand? The problem with your approach is that you pretty much have to customise that for every table, since the filed definitions will vary...

BTW, which DBMS and library are you using?

Author

Commented:
Hi jkr, it's more complicated than that - I'm asking a more basic question than what I need - I am using Postgres.  Basically, I will use this with streams and to do synchronization between databases (sort of).  I have database objects smart enough to know how to save their own data, etc....  I've started by using XML - I'm hoping it's not too slow with all the text and extra lengths because of tag identifiers, etc.  Do you know of a library that can zip as I write to a file?

Mike
evilrixSenior Software Engineer (Avast)
SILVER EXPERT

Commented:
>> Basically, I will use this with streams and to do synchronization between databases (sort of).
Why don't you just use database replication?
http://www.postgresql.org/about/news.233

Author

Commented:
I can't replicate the database tables - some tables will need to have minor differences (private data not to be shared with others)....
evilrixSenior Software Engineer (Avast)
SILVER EXPERT

Commented:
BTW: Since you are using Postgres, why did you post to the MySQL Zone? There is a Postgres zone, I'm sure you would have been better off posting to there (I know MySQL but not Postgres!).
http://www.experts-exchange.com/Database/PostgreSQL/
evilrixSenior Software Engineer (Avast)
SILVER EXPERT

Commented:
>> I can't replicate the database tables - some tables will need to have minor differences (private data not to be shared with others)....
Trying to synchronize manually it not going to be a trivial task. Writing the code to read from one and write to another is pretty simply. The problem is going to be managing concurrency. Have you given any thought to how you are going to do this?

evilrixSenior Software Engineer (Avast)
SILVER EXPERT

Commented:
BTW: You might want to consider using ADO.Net for this.
http://en.wikipedia.org/wiki/ADO.NET

Author

Commented:
evilrix - it's not full synchronization... It's a simplification - and it took me about a year to figure out how to do it (to answer your question about whether or not I thought about how I was going to do this)... . I've also got 3 years of database objects coded that use ODBC... can't switch to nicer newer APIs this late in the game unfortunately.... :-)   Thanks for your ideas & suggestions.

Mike
evilrixSenior Software Engineer (Avast)
SILVER EXPERT

Commented:
To be honest I'm a little at a lose as to what you actual question is then? You just want to know how to open and write a file stream in C++? See below.

http://www.cplusplus.com/doc/tutorial/files.html
http://www.cplusplus.com/reference/iostream/
#include <fstream>
 
int main()
{
	std::fstream fs("c:\\temp\\test.txt", std::ios::out | std::ios::in | std::ios::app);
	fs << "Hello World" << std::endl;
}

Open in new window

Author

Commented:
Hi evilrix:

I want to know how to link up functions - so I can pipe a stream one way or the other (kind of like OpenSSL BIO objects in case you're familiar with them)..   It may be a silly question, but I've just never piped one function's input stream to another.... I just need some background and maybe if you know about a compression "stream" that I can write to as well....

Mike
Senior Software Engineer (Avast)
SILVER EXPERT
Commented:
>> so I can pipe a stream one way or the other
C++ IO streams aren't really designed to be piped to each other. You'll have to read from one stream and write to another. A stream is really just a fancy name for a common interface that wraps various data buffers (whether it's file or memory or other) and presents it in a uniform and extensible way.

>> and maybe if you know about a compression "stream" that I can write to as well....
If you are serious about this you should take a look at boost iostreams. They allow you to write your own streams based upon the standard ones (they expose the same interface so they can be used as drop in replacements) but with the added advantage that you can implement filers and attach them to the stream (so a filer could zip as you write and unzip as you read). There's a fair bit of reading to do but since this project has been 3 years in the making a few more hours RTM won't hurt :)

http://www.boost.org/libs/iostreams/doc/
http://www.boost.org/libs/iostreams/doc/classes/bzip2.html
http://www.boost.org/libs/iostreams/doc/classes/gzip.html
http://www.boost.org/libs/iostreams/doc/classes/zlib.html

Author

Commented:
exactly what I was looking for!  Thank you very much!  Many thanks!  :-)  (I think that's the one that a colleague had used a few years back)...
evilrixSenior Software Engineer (Avast)
SILVER EXPERT

Commented:
One last thing. Building Boost (on Windows) can be a painful experience since it uses its own proprietary build system (BJam). Fortunately, you can download precompiled binaries for Windows from the following site: http://www.boost-consulting.com/products/free

I hope this helps and good luck.

-Rx.

Author

Commented:
Thanks for the luck - I've got a serious load of work to do and only a few months to get it all done & tested....  It's good not to feel so alone doing it....  And thanks again.

Author

Commented:
Oh and best to you as well.
evilrixSenior Software Engineer (Avast)
SILVER EXPERT

Commented:
>> It's good not to feel so alone doing it...
Always here to help -- well, unless my wife is giving me grief for being at the PC "again" :)
Seriously though, if you get stuck just shout.

>> Oh and best to you as well.
And the best to you too my friend.

-Rx.

Author

Commented:
>> Always here to help -- well, unless my wife is giving me grief for being at the PC "again" :)
Haha - Don't I hear ya....  :-)  
Cheers,
Mike

Explore More ContentExplore courses, solutions, and other research materials related to this topic.