Complex Scenario - SAS

Hi all,

I have a text file on Server A and I need to SCP it to server B and create a SAS dataset. I have the code built for this functionality. But the file is around 80 million records and for every run I need only a subset of the text file.

The current process is not efficient as it is pull all the file for every run and agin doing a subset of the sas dataset.

Is there any way where I can only pull a subset of data directly from the text file to server B?

I thought of creating a Shell script where it will create a subset there in server A and then pull the subset every time. wanted to know if there is any more efficient way of doing this.

Please help.

Thanks in advance.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

d507201Database Marketing ConsultantCommented:
At the 2011 SGF I presented a paper on using SAS to move data between servers and the paper has been posted on this site.  Under the Articles tab, search for SAS.

If both servers have SAS and you can set up a client-server relationship between the two servers then a combination of remote compute services and data transfer services or remote library services will easily do the job.  

If SAS isn't on one of the servers, but you do have FTP or SFTP on both servers then SAS has access methods that will workl with those.  (9.1 supports only FTP but 9.2 and up support both FTP and SFTP.)  It's not in the paper but my powerpoint deck shows a code example to read a text file and subset it while moving it via FTP.  It works the same way under SFTP.  Let me know if you need to see the powerpoint.

Read the article and post here if you need more help.
arukuAuthor Commented:
Thanks for the comments d507201. Can I see the power point as that will help me with resolving this issue.
d507201Database Marketing ConsultantCommented:
Slide 14 is the one that talks about FTPing a text file and subsetting at the same time.  

Slides 18 and 19 are about SCP.  19 has examples of using the X statement to run SCP from within a SAS program.    

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Aloysius LowCommented:
i would say subsetting the data before transferring / accessing directly from server B is the most efficient approach... if you try to access the data on server A directly from server B to subset the data, inevidently you are pulling all records to server B before the subset takes place...

otherwise, you have to consider a change in approach to generate the file on server A - why is the data all inside 1 file? for e.g. can the new records be created in a new file? could you do a grep to get only the date/time of the records you one and write the output into another file to be read/transferred?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.