I have a text file on Server A and I need to SCP it to server B and create a SAS dataset. I have the code built for this functionality. But the file is around 80 million records and for every run I need only a subset of the text file.
The current process is not efficient as it is pull all the file for every run and agin doing a subset of the sas dataset.
Is there any way where I can only pull a subset of data directly from the text file to server B?
I thought of creating a Shell script where it will create a subset there in server A and then pull the subset every time. wanted to know if there is any more efficient way of doing this.
Thanks in advance.