I have a flat file which having around 1-3gb data , and before do import in the database, i need some clean up so that when I do import only a new clean data get imported. Below is how we doing it in the database via T-sql but since the database cannot accept any downtime and the select query below always takes very very long to complete and very often block insert to the database, we decided to move this task to text file string manipulation.
do anyone know how to begin with Powershell command to output new file without having any values below within a rows ?
Powershell command select-string will do this job but I'm clueless how to write the code for the clean up.
AGT_domainname = '' OR
AGT_username LIKE 'SELkry' OR
AGT_username LIKE 'testuser%' OR
AGT_username LIKE 'siteminder' OR
AGT_username LIKE 'CN=MYR%' OR
AGT_username LIKE 'CN=MYRTEST%' OR
AGT_username LIKE 'selypz' OR
AGT_username LIKE 'selyyz' OR
AGT_domainname LIKE 'Monitoring External' OR
AGT_domainname LIKE 'onitoring Internal' OR
AGT_domainname LIKE 'Test Domain' OR
AGT_domainname LIKE 'TXTAPP SE' OR
AGT_domainname LIKE 'TXTAPP COM' OR
AGT_domainname LIKE 'TXTAPP NET' OR
AGT_domainname LIKE 'TXTAPP FED' OR
AGT_eventid = '7' OR
AGT_eventid = '8' OR
AGT_eventid = '9'