Automate data import process

Posted on 2014-09-07
Last Modified: 2016-02-17
I'm using the Salesforce platform and I currently have multiple flat files that need to be imported daily. Currently, I manually import the files using data loader. I am doing very simple imports to Accounts and Opportunities objects in Salesforce.

The process is two prong. First, we have a process that exports the data from another database to a flat file (csv) and copies it over to a FTP site. Then I will grab the flat file (csv) from the FTP site and load into Salesforce.

I am looking at ways to automate this process with no user intervention. I am looking at the automated pull approach or an http request / response. Is there any Salesforce examples of how each of these processes would be implemented? I'm sure this is probably a pretty common problem. I don't want to do any command line scripting process and using a middleware solutions like Informatica Cloud isn't on the table  at the present time.

I haven't had a ton of integration experience, but as I understand it I have the following choices if I dont want to license a tool, such as Informatica Cloud. I could a) write an apex rest service that would accept a POST with the new data, or b) write an inbound email handler that accepted csv attachments or c) have a scheduled apex job that made an outbound rest/soap call to a rest/soap service on your source system.   I am not very familiar with REST, but option "a" seems like a clean solution, but an external web service would have to be written to POST to an APEX rest service that can accept the POST, correct?  Option "b" seems like it wouldn't be very viable based on potential size issue of the flat files and parsing the csv file.  Option "c" seems reasonable too, but again, there would need to be an external REST/SOAP rest service that I can make a call to that would need to be built.

Being completely new to integration development (REST), does anyone have any examples they can point me to that demonstrate how to accomplish options "a" and "b"?  

Has anyone replaced the dataloader manual loading process in Salesforce with an automated process using REST or SOAP?  I'm looking for any guidance on how to accomplish something like this.

Any examples of how to implement this functionality would be immensely helpful.

Thanks in advance for any help.
Question by:-Dman100-
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
LVL 33

Expert Comment

ID: 40309904
We're using the Salesforce platform too.
We also have a variety of online sources that create leads.  Sometimes these are services that have internal built-in integration with salesforce, but often are not integrated.

For example, we are subscribed to a data feed that has daily lists of unqualified leads relevant to our business.

We use the Itduzzit service to automatically get the data feed, enrich it with information from the crunchbase API, create new leads in Salesforce, and assign the leads to different sales people based on some internal criteria.

Another usage we do is to get accounts that are in a certain status in Salesforce (entitled to service) and automatically provision them to our support platform via REST API.

The $59 a month plan has unlimited integrations, it supports FTP connectors, and I highly recommend it.  
It can reach out via FTP, get a CSV file, and create your SF accounts.

Author Comment

ID: 40311085
Hi Shalomic,

Thanks for your reply and information.  How does Itduzzit work?  Is there any programming required?  For example, how does it read a CSV file on a FTP site?  

Do you run into any governor limits (API calls, etc.?)  Can it handle bi-directional processing?  Can it be triggered real-time or just on a scheduled run?

Thanks for your help and the information!!
LVL 33

Accepted Solution

shalomc earned 500 total points
ID: 40313290
> How does Itduzzit work?  
Create a free trial and see for yourself.

>Is there any programming required?
You need to understand programming but you don't write code.

> how does it read a CSV file on a FTP site?
You will probably create a custom "duzzit" (that's how they call integration units) that uses the FTP connector to fetch a file. There is a built-in CSV parser, just needs mapping.

> Do you run into any governor limits (API calls, etc.?)
The number of calls depends on your commercial plan.

> Can it handle bi-directional processing?

> Can it be triggered real-time or just on a scheduled run?

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
DirSync to Office 365 5 67
Moving oracle db server (10g, 11.2 version)  to Azure cloud oracle version 12C 1 221
Office365 9 137
Web Service for a door knocking sales campaign 9 133
Shadow IT is coming out of the shadows as more businesses are choosing cloud-based applications. It is now a multi-cloud world for most organizations. Simultaneously, most businesses have yet to consolidate with one cloud provider or define an offic…
Big data transfers via information superhighways require special attention and protection. Learn more about the IT-regulations of the country where your server is located. Analyze cloud providers and their encryption systems for safe data transit. S…
This Micro Tutorial will explain how to export DynamoDB tables in Amazon Web Services.
Internet Business Fax to Email Made Easy - With  eFax Corporate (, you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…

735 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question