Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Access/SQL ETL (Extract Transfer Load)

Posted on 2014-02-05
3
Medium Priority
?
735 Views
Last Modified: 2014-02-06
Hi

I have an Access database that has tables that are linked to SQL.
On a daily basis the data from one of the tables (circa 40000 rows) has to be
be downloaded into Access, edited and uploaded again.

I was told that the process is referred to as ETL (Extract Transfer Load)
where the linked table is converted into an Access table, the SQL index column is dropped.
Then data is edited and added in Access and finally re-uploaded.

What is the best way to do this? I assume that a mix of VBA and queries will be necessary
0
Comment
Question by:Murray Brown
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
3 Comments
 
LVL 51

Expert Comment

by:Gustav Brock
ID: 39838151
40000 rows isn't that much, so just create a link (attach) the table via ODBC and manipulate the records. Much easier than importing, deleting, and refilling the table.

/gustav
0
 
LVL 24

Accepted Solution

by:
Bitsqueezer earned 2000 total points
ID: 39838345
Hi,

I would also recommend to use Gustav's solution. The reason is that a user would normally edit only one record at a time so the risk of concurrency is relatively small (but you need of course a 100% online connection to the backend). If one user edit the same record as another one, Access handles that automatically and warns you if the record on the server is no longer the same as the own one - no code needed (but you should add a column of type "timestamp" to any table which is edited by Access directly).

If you really need to edit an "offline" version you would need to handle the concurrency on your own. That means: If the user edit 10 records where another user also edited and deleted 6 of them with his own frontend then you need to have a way to handle that when you upload the changes of user 1.

You could use an Access table for that and convert that and so on - but there is a much easier way to do that especially because you have only few records (40,000 is really not much): Use ADO offline recordsets. The trick is simple: Create an ADO recordset and close the connection of the recordset after downloading it. ADO offers a bunch of properties and methods for batch processing so you can easily let the user edit that recordset with a normal Access form by assigning the recordset directly to the form recordset (which can be either DAO or ADO). There are some differences in handling, i.e. the ID column cannot get a new value at the time the user edit it, ADO automatically uses a replacement ID (negative number as far as I remember) and replaces all new record IDs later when the changes are submitted to the server. When the user clicks a "Submit" button, your code can then reenable the connection to the server and use the methods of ADO which offers for example a conflict list that tells you which record was successfully updated and where are which kind of conflicts (like "other user changed that" or "record was already deleted" and so on) and offers methods to handle these conflicts. You only need to program that in VBA because you must say what to do in which conflict case.

Using this method you do not need any local table, no conversion (as you are working with the server's datatypes in the ADO recordset) and no complicate compare methods as that is all already contained in ADO.
By the way: .NET uses a similar technique, only calls that dataset which can hold a bunch of tables but the way is the same: Working always offline, working with the data in memory and having a complete set of conflict methods. If you have learned the batch process handling in ADO you will not have any problem to understand the same in ADO.

Please use your favourite search machine to search for the technical documentation about batch processing in ADO for further details as that would be a lot to explain in detail here.

Cheers,

Christian
0
 

Author Closing Comment

by:Murray Brown
ID: 39838510
Thanks
0

Featured Post

Get your Conversational Ransomware Defense e‑book

This e-book gives you an insight into the ransomware threat and reviews the fundamentals of top-notch ransomware preparedness and recovery. To help you protect yourself and your organization. The initial infection may be inevitable, so the best protection is to be fully prepared.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Microsoft Access is a place to store data within tables and represent this stored data using multiple database objects such as in form of macros, forms, reports, etc. After a MS Access database is created there is need to improve the performance and…
Traditionally, the method to display pictures in Access forms and reports is to first download them from URLs to a folder, record the path in a table and then let the form or report pull the pictures from that folder. But why not let Windows retr…
With Microsoft Access, learn how to start a database in different ways and produce different start-up actions allowing you to use a single database to perform multiple tasks. Specify a start-up form through options: Specify an Autoexec macro: Us…
Add bar graphs to Access queries using Unicode block characters. Graphs appear on every record in the color you want. Give life to numbers. Hopes this gives you ideas on visualizing your data in new ways ~ Create a calculated field in a query: …

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question