How can I use shell script to read and drop duplicate records in a flat file?

Hi all,
My environment DB2 UDBv8 going on to v9, Linux OS. What I am looking for is a way to read a flat file created by a script and remove duplicate records. The idea is to read the file and write to a new file only unique records.

Here is a sample of the contents of the input file:

REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INTAKE_PART ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INVESTIGATION ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INVESTIGATION ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INVESTIGATION_CNTC ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INVESTIGATION_CNTC ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INVESTIGATION_CNTC ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MEDICAL_PROFILE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MEDICATIONS ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MENTAL_HEALTH_PROFILE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MPHONE_FRST ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MPHONE_FRST ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MPHONE_LST ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.ONCALL_SCHEDULE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.ORG_WORKER_ROLE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PAN_TEXT_EVENT ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PAN_TEXT_EVENT ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PERSON_AKA ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PROVIDER_ORG ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PROVIDER_ORG ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PROVIDER_ORG ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PROVIDER_PART ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PROVIDER_PART_STAT ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.RELATIONSHIP ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.RELATIONSHIP ALLOW WRITE ACCESS;


What I want to see in the new file is just only 14 records in the new file:

REORG INDEXES ALL FOR TABLE DB2ADMIN.EPISODE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INTAKE_PART ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.INVESTIGATION_CNTC ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MEDICAL_PROFILE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MEDICATIONS ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MENTAL_HEALTH_PROFILE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.MPHONE_LST ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.ONCALL_SCHEDULE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.ORG_WORKER_ROLE ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PAN_TEXT_EVENT ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PERSON_AKA ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PROVIDER_ORG ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.PROVIDER_PART ALLOW WRITE ACCESS;
REORG INDEXES ALL FOR TABLE DB2ADMIN.RELATIONSHIP ALLOW WRITE ACCESS;

Can someone help me to solve this problem? Again, I will like to do it in a shell script of some sort and be able to incorporate it into the script that creates the original file.

Thanks
Start Free Trial