Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


Complex Scenario - SAS

Posted on 2011-10-05
Medium Priority
Last Modified: 2013-11-16
Hi all,

I have a text file on Server A and I need to SCP it to server B and create a SAS dataset. I have the code built for this functionality. But the file is around 80 million records and for every run I need only a subset of the text file.

The current process is not efficient as it is pull all the file for every run and agin doing a subset of the sas dataset.

Is there any way where I can only pull a subset of data directly from the text file to server B?

I thought of creating a Shell script where it will create a subset there in server A and then pull the subset every time. wanted to know if there is any more efficient way of doing this.

Please help.

Thanks in advance.
Question by:aruku
  • 2

Expert Comment

ID: 36919414
At the 2011 SGF I presented a paper on using SAS to move data between servers and the paper has been posted on this site.  Under the Articles tab, search for SAS.

If both servers have SAS and you can set up a client-server relationship between the two servers then a combination of remote compute services and data transfer services or remote library services will easily do the job.  

If SAS isn't on one of the servers, but you do have FTP or SFTP on both servers then SAS has access methods that will workl with those.  (9.1 supports only FTP but 9.2 and up support both FTP and SFTP.)  It's not in the paper but my powerpoint deck shows a code example to read a text file and subset it while moving it via FTP.  It works the same way under SFTP.  Let me know if you need to see the powerpoint.

Read the article and post here if you need more help.

Author Comment

ID: 36919493
Thanks for the comments d507201. Can I see the power point as that will help me with resolving this issue.

Accepted Solution

d507201 earned 2000 total points
ID: 36919684
Slide 14 is the one that talks about FTPing a text file and subsetting at the same time.  

Slides 18 and 19 are about SCP.  19 has examples of using the X statement to run SCP from within a SAS program.    
LVL 14

Expert Comment

by:Aloysius Low
ID: 36927856
i would say subsetting the data before transferring / accessing directly from server B is the most efficient approach... if you try to access the data on server A directly from server B to subset the data, inevidently you are pulling all records to server B before the subset takes place...

otherwise, you have to consider a change in approach to generate the file on server A - why is the data all inside 1 file? for e.g. can the new records be created in a new file? could you do a grep to get only the date/time of the records you one and write the output into another file to be read/transferred?

Featured Post

[Webinar] Database Backup and Recovery

Does your company store data on premises, off site, in the cloud, or a combination of these? If you answered “yes”, you need a data backup recovery plan that fits each and every platform. Watch now as as Percona teaches us how to build agile data backup recovery plan.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In today's business world, data is more important than ever for informing marketing campaigns. Accessing and using data, however, may not come naturally to some creative marketing professionals. Here are four tips for adapting to wield data for insi…
Among the most obnoxious of Exchange errors is error 1216 – Attached Database Mismatch error of the Jet Database Engine. When faced with this error, users may have to suffer from mailbox inaccessibility and in worst situations, permanent data loss.
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…
Despite its rising prevalence in the business world, "the cloud" is still misunderstood. Some companies still believe common misconceptions about lack of security in cloud solutions and many misuses of cloud storage options still occur every day. …

581 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question