database synchronization between traditional RDMS and NoSQL


any  tools/idea you all use to synchronization between traditional RDMS and NoSQL in MS SQL and maria DB and noSQL?
marrowyungSenior Technical architecture (Data)Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David FavorLinux/LXD/WordPress/Hosting SavantCommented:
This almost always requires custom code, as each combination will have complex indexing (SQL side) which must be normalize going to simple indexing (no-SQL side).
marrowyungSenior Technical architecture (Data)Author Commented:
how about ETL and replication server like oracle goldengate, sybase replication serer ?
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Same answer. You're moving from complex SQL indexing to simple non-SQL indexing, hashed lookups.

Each of these types of moves tend to be unique in their design.
Get Blueprints for Increased Customer Retention

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

marrowyungSenior Technical architecture (Data)Author Commented:
so can't do much on this ! no tools for it, right?
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
No tools for this. You'd write custom tools designed like this.

1) Open connections to database + no-SQL store.

2) Generate your custom SQL SELECT calls.

3) Convert retrieved data into one or more no-SQL writes.

4) Do all your writes.

5) Rinse + repeat for all SQL data tables.

6) Close all your connections.

When I have a client requesting this type of work, I just use PERL to quickly whip up a one-time, throw-away script.
marrowyungSenior Technical architecture (Data)Author Commented:
you are PERL programmer ?
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
I write PERL code pretty much every day, for projects + command line tools I use with my private hosting clients.

I've been using PERL since... 1994, so I'm guessing I qualify as a PERL programmer.

I only work on code for sites I host though + I'm very expensive.

Likely your best bet is to open an EE Gig + someone can assist you with your project.
marrowyungSenior Technical architecture (Data)Author Commented:
"I only work on code for sites I host though + I'm very expensive."

ahahha, a lot of people here say the same thing.

other than PERL,  c# can do that too ?
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
All languages are just languages. So any language will work.

That said, if I had a client suggest c# I'd try to contain my laughter... probably couldn't...

I'd figure 10x to 100x the time for c# instead of PERL.

Maybe longer, as your first step will be getting c# to actually work under Linux... as I'm hoping you're using Linux for your project.

If you're imagining Microsoft as your target OS... whew... add 10x to every minor daily task.

You're going to require a massive budget for your project is you're using non-Linux based solution. covers the LAMP Stack code versions my team currently uses for projects.

I always use Ubuntu for host + container OS, then run LXD at the machine level, where sites/projects all live in their own LXD containers.

You may find this helpful also, a you can clone a production container into a dev container in a few seconds.

This also allows you to deploy a dev site as a production site in roughly 30 seconds.
marrowyungSenior Technical architecture (Data)Author Commented:
"Maybe longer, as your first step will be getting c# to actually work under Linux... as I'm hoping you're using Linux for your project."

excellent  !

but I heard that Tungsten replicator 3.0 can replicate from MySQL / oracle to NoSQL like Hadoop, right? it seems nearly real time, right?
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
It's not an issue of speed.

The issue is converting complex indexes (SQL) to simple/single/hashed indexes (no-SQL).

Once you start this process once, you'll understand the hurdles involved.

Just pick a database + table + start your conversion process. Pretty simple + sometimes time consuming to complete.

Many consideration is code accessing your non-SQL data will have to be able to lookup data + use it as you envision.

One other option, I'm doing this for a client now, is keeping complex SQL + being able to do Google-ish type searches on various data.

To implement this, I'm using a very simple row structure, basically no-SQL, with a primary key + FTS (Full Text Search) type indexing on top of a column of data tags - ad-hoc keywords + keyphrases which can be anything.

v covers details of how this works.

Since you haven't mentioned how your no-SQL application will work, difficult to guess anymore about how to assist with your conversion.

Based on your questions, likely best if you hire someone to assist you with your project.

A good designer may look at your end goal + say... "Geez, you can use no-SQL + you'll end up writing far more code + have a barely maintainable system or you can just add an FTS index to your existing SQL database + write code to do FTS lookups."

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
marrowyungSenior Technical architecture (Data)Author Commented:
v covers details of how this works.
oh, by using MariaDB ?
you mean Maria DB can do a lot in this area?

"Based on your questions, likely best if you hire someone to assist you with your project."
you mean Database developer ?
marrowyungSenior Technical architecture (Data)Author Commented:
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.