Link to home
Start Free TrialLog in
Avatar of thready
thready

asked on

Write database table to a stream and read stream back into other equivalent database table

Hi Experts,
I would like to design a fast algorithm that can take items in a database table and spit them out to a stream.  I'm new with stream programming - but I know there is a way to write to a file as a stream or treat sockets as the stream.  Can someone point out how to set this up?   And how would I delimit my records?

Many thanks,
Mike
Avatar of jkr
jkr
Flag of Germany image

Well, wouldn't it be easier to dump the table to a file and then reload it instead of doing that by hand? The problem with your approach is that you pretty much have to customise that for every table, since the filed definitions will vary...

BTW, which DBMS and library are you using?
Avatar of thready
thready

ASKER

Hi jkr, it's more complicated than that - I'm asking a more basic question than what I need - I am using Postgres.  Basically, I will use this with streams and to do synchronization between databases (sort of).  I have database objects smart enough to know how to save their own data, etc....  I've started by using XML - I'm hoping it's not too slow with all the text and extra lengths because of tag identifiers, etc.  Do you know of a library that can zip as I write to a file?

Mike
>> Basically, I will use this with streams and to do synchronization between databases (sort of).
Why don't you just use database replication?
http://www.postgresql.org/about/news.233

Avatar of thready

ASKER

I can't replicate the database tables - some tables will need to have minor differences (private data not to be shared with others)....
BTW: Since you are using Postgres, why did you post to the MySQL Zone? There is a Postgres zone, I'm sure you would have been better off posting to there (I know MySQL but not Postgres!).
https://www.experts-exchange.com/Database/PostgreSQL/
>> I can't replicate the database tables - some tables will need to have minor differences (private data not to be shared with others)....
Trying to synchronize manually it not going to be a trivial task. Writing the code to read from one and write to another is pretty simply. The problem is going to be managing concurrency. Have you given any thought to how you are going to do this?

BTW: You might want to consider using ADO.Net for this.
http://en.wikipedia.org/wiki/ADO.NET
Avatar of thready

ASKER

evilrix - it's not full synchronization... It's a simplification - and it took me about a year to figure out how to do it (to answer your question about whether or not I thought about how I was going to do this)... . I've also got 3 years of database objects coded that use ODBC... can't switch to nicer newer APIs this late in the game unfortunately.... :-)   Thanks for your ideas & suggestions.

Mike
To be honest I'm a little at a lose as to what you actual question is then? You just want to know how to open and write a file stream in C++? See below.

http://www.cplusplus.com/doc/tutorial/files.html
http://www.cplusplus.com/reference/iostream/
#include <fstream>
 
int main()
{
	std::fstream fs("c:\\temp\\test.txt", std::ios::out | std::ios::in | std::ios::app);
	fs << "Hello World" << std::endl;
}

Open in new window

Avatar of thready

ASKER

Hi evilrix:

I want to know how to link up functions - so I can pipe a stream one way or the other (kind of like OpenSSL BIO objects in case you're familiar with them)..   It may be a silly question, but I've just never piped one function's input stream to another.... I just need some background and maybe if you know about a compression "stream" that I can write to as well....

Mike
ASKER CERTIFIED SOLUTION
Avatar of evilrix
evilrix
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of thready

ASKER

exactly what I was looking for!  Thank you very much!  Many thanks!  :-)  (I think that's the one that a colleague had used a few years back)...
One last thing. Building Boost (on Windows) can be a painful experience since it uses its own proprietary build system (BJam). Fortunately, you can download precompiled binaries for Windows from the following site: http://www.boost-consulting.com/products/free

I hope this helps and good luck.

-Rx.
Avatar of thready

ASKER

Thanks for the luck - I've got a serious load of work to do and only a few months to get it all done & tested....  It's good not to feel so alone doing it....  And thanks again.
Avatar of thready

ASKER

Oh and best to you as well.
>> It's good not to feel so alone doing it...
Always here to help -- well, unless my wife is giving me grief for being at the PC "again" :)
Seriously though, if you get stuck just shout.

>> Oh and best to you as well.
And the best to you too my friend.

-Rx.
Avatar of thready

ASKER

>> Always here to help -- well, unless my wife is giving me grief for being at the PC "again" :)
Haha - Don't I hear ya....  :-)  
Cheers,
Mike