C#: How do you read a text file into a array removing dupilcate lines from a Second Log file


I have a list of URL's in a file that I need to process
and a List of  finished URL's in another Log file
after I've finished with a URL I write / append it to the Log  File

How do check the current URL isn't in the Log File therefore not already processed

using System;
using System.IO;
using System.Linq;

public class Program
	public static void Main()
		    string LogFile = System.Environment.CurrentDirectory + "\\Log.txt";
            var URLFromLog = File.ReadAllLines(LogFile).Select(l => l.Trim());
            var URLFromURLs = File.ReadAllLines(@"Path\To\URLs.txt").Select(l => l.Trim()); // This File may contain Duplicates and changes each time the program runs
		// Only Add to URLs if unique and not already in the Log File

// This doesn't look right but unsure what it should be
            var URLs = URLFromLog.Distinct().Concat(URLFromURLs).ToList();
		foreach (string sURL in URLs)
			    var URL = sURL;
			// Do Stuff with URL 
                File.AppendAllText(LogFile, sURL + Environment.NewLine);

Open in new window

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Chris StanyonWebDevCommented:
Have a look at a HashSet. Very fast and ensures uniqueness:

var logFile = Path.Combine(Environment.CurrentDirectory, "Log.txt");
var logFileEntries = new HashSet<String>(File.ReadAllLines(logFile).Select(l => l.Trim()));
var urls = File.ReadAllLines(@"Path\To\URLs.txt").Select(l => l.Trim());

using (var writer = new StreamWriter(logFile))
    foreach (var entry in urls)
        if (logFileEntries.Add(entry))

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
trevor1940Author Commented:
Thank you
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
.NET Programming

From novice to tech pro — start learning today.