How to read files into a Dictionary without Duplicates Vlaues from a directory using C#?

Hello Experts,

I have a requirement that I need to read all the text files from a particular directory (C:\myProject).The files contains key value pairs separated by space. The commented lines begin with the value '#' and it should be ignored. The key can be ends with '-" and '-' should not be included in the key. I need to print all the keys without any duplicates. Also the performance is very important.


Tom 575858
Jerry 4444
A 123
#Hello World
GA- 1234


A  123
B  456
C  567

Expected Result:

C B A Jerry Tom FA GA

I am looking for a good solution that  meets my requirement. Thanks in advance for your suggestions and help.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

kaufmed 👽Commented:
Depending on how many files you'll be working with, you might try using the new Parallel class:


using System;
using System.Collections.Concurrent;
using System.IO;
using System.Threading.Tasks;

namespace _28689280
    class Program
        static ConcurrentDictionary<string, byte> buffer = new ConcurrentDictionary<string, byte>();

        static void Main(string[] args)
            ParallelOptions options = new ParallelOptions() { MaxDegreeOfParallelism = 4 };

            Parallel.ForEach(Directory.EnumerateFiles(".", "*.txt"), options, ProcessFile);


        async static void ProcessFile(string path)
            using (FileStream stream = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.None, 4096, FileOptions.SequentialScan | FileOptions.Asynchronous))
                using (StreamReader reader = new StreamReader(stream))
                    while (!reader.EndOfStream)
                        string line = await reader.ReadLineAsync();

                        if (!line.StartsWith("#"))
                            string key = line.Substring(0, line.IndexOfAny(new[] { ' ', '-' }));

                            if (buffer.TryAdd(key, 0))
                                Console.Write(key + " ");

Open in new window

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.