Solved

Efficient adjusting of an increment variable to append sets

Posted on 2009-07-15
4
634 Views
Last Modified: 2013-11-16
I have two datasets with the same variables on them that I'd like to append. The only problem is that they both have an ID variable that starts at 1 and increments for each record. When I append them, I want to take the last value of ID in the first dataset and add it to the values of ID in the second dataset, so that when they're appended I have a unique ID variable for each entry that increases as you go down the set.

The main issue is that both of these datasets are big - as in "takes 5 minutes to just run a basic data step on them" big. So while I can think of a few ways to do what I want, they all involve processing one or other of the datasets multiple times, resulting in a slow process. Is there a quick way to do it?

(Small edit to add info: generally, the second dataset is smaller than the first, so it's a little quicker to access as well if that affects the solution.)
0
Comment
Question by:Confusing
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 14

Accepted Solution

by:
Aloysius Low earned 250 total points
ID: 24867569
hi,

what i can think of is to run a data step, but instead of using ID from either table, use _N_ instead

i.e.
data C;
  set A (drop = ID) B (drop = ID);
  ID_NEW = _N_;
run;

however, this assumes that the ID on both tables are in running order.
0
 
LVL 4

Assisted Solution

by:wigmeister
wigmeister earned 250 total points
ID: 24878036
I would probably do this:

Take your largest dataset and get the number of obs in that dataset:
data _null_;
if 0 then set test1 nobs=nobs;
call symputx(nobs,nobs);  **symput
stop;
run;

Then step through your smaller dataset and increment ID:
data test2;
  set test2;
id = id + &nobs
run;

Put them together:
proc append base = test1 data=test2;
run;

You will probably have to adjust the way &nobs is being used in the equation, but you get the idea.  This is basically doing the same tihng that lowaloysius suggested, except it processes the smaller dataset (test2) twice, instead of the 'set' which will process both once.

I still think there might be a more efficient way somehow using macros, but it hasn't come to mind yet.  If I think of anything I will post back.
0
 
LVL 1

Author Comment

by:Confusing
ID: 24901530
I actually found a solution myself that is about as efficient as lowaloysius', but for the excellent alternative methods I'm going to split the points between the two of you.
0
 
LVL 14

Expert Comment

by:Aloysius Low
ID: 24905360
Out of curiosity, what is it that you have got?
0

Featured Post

Get Database Help Now w/ Support & Database Audit

Keeping your database environment tuned, optimized and high-performance is key to achieving business goals. If your database goes down, so does your business. Percona experts have a long history of helping enterprises ensure their databases are running smoothly.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When table data gets too large to manage or queries take too long to execute the solution is often to buy bigger hardware or assign more CPUs and memory resources to the machine to solve the problem. However, the best, cheapest and most effective so…
A company’s centralized system that manages user data, security, and distributed resources is often a focus of criminal attention. Active Directory (AD) is no exception. In truth, it’s even more likely to be targeted due to the number of companies …
Video by: Steve
Using examples as well as descriptions, step through each of the common simple join types, explaining differences in syntax, differences in expected outputs and showing how the queries run along with the actual outputs based upon a simple set of dem…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question