Serious debugging problem urgent....

You need to look earlier in your code, somewhere before you call the process() method.

If user1 and user2 are both being affected by a command (eg. add()), then they must both be referencing the same object, so:

          User user1 = (User)users[i];
          User user2 = (User)users[j];

must mean that the users array contains references to the same object.  You need to check how the users array is being created.

When i throw in statements to give me user1s id and user 2s id here, it gives me the correct ids:

          User user1 = (User)users[i];
          System.out.println("User1s id is" + user1.getUserId());
          User user2 = (User)users[j];
          System.out.println("User2s id is" + user2.getUserId());

Every iteration prints out the correct user id.

The users array is created like this:

Object[] users = userArrayList.toArray();

Is that causing a problem because i'm now referencing an array instead of an arraylist?

Ah.  The problem is not with the users, it is with the msdSimilarUsers attribute within them.

Either you have declared this as static (so all users share the same msdSimilarUsers data) or when you set up the msdSimilarUsers attribute, you may be adding references to the same object to each of the User objects.

I'd check to see if this is static first ;-)

Here's the user class where i have the arraylist of sim users setup:

class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;

    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;
    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }

}

I do make a static declaration at the start of my program:

class ***

  private static ArrayList userArrayList = new ArrayList();

Would that be the problem?

I did some further debugging and what happens exactly on the 1st iteration is that user2 is added to user1s and user2s arraylist of similarUsers.

On the next iteration it adds another user into both arraylists.

Then it moves on from user1 and user1 now becomes user2 and user2 becomes3 etc as each step takes place.

However thats where the funny stuff starts to happen as it keeps populating the original users arrylist with similar users even though that user is no longer being used in the comparison!

I also tested each users arraylist of similar users and they all contain the exact same users(including replicas of certain users!)

Heres all the relevant code if anybody wants to have a look:

import java.io.*;
import java.util.*;

class FileParser {

  private static ArrayList userArrayList = new ArrayList();

  public static void main(String[] args) throws Exception {

    FileReader fr =
      new FileReader("C:/dataset1.txt"); // to open the file
      BufferedReader inFile = new BufferedReader(fr);     // to read the file
      String line;     // String variable used to hold each line of the file
      StringTokenizer tokenizer;     // string tokenizer to parse each file line
      ArrayList userArrayList = new ArrayList();
      ArrayList msdSimilarUsers = new ArrayList();
      ArrayList pearsonsSimilarUsers = new ArrayList();
      ArrayList fullArrayList = new ArrayList();
      int id = 0;
      int index = 0;     // Index that we increment each time through the loop
      //User user = null;
      int movieId;     // value to hold movie id
      float ratings;     // value to hold movie ratings
      ArrayList pair = null;
      while((line = inFile.readLine()) != null) {
        tokenizer = new StringTokenizer(line);     // tokenize the file line
        // now add the user id to the user id array
        id = Integer.parseInt(tokenizer.nextToken());
        // initialize the movie id variable
        movieId = Integer.parseInt(tokenizer.nextToken());
        // initialize the cd ratings variable
        ratings = Float.parseFloat(tokenizer.nextToken());
        // Check out if the user has already been added to the list
        Iterator iter = userArrayList.iterator();
        User user = null;
        while (iter.hasNext()) {
          User tmp = (User)iter.next();
          if (tmp.getUserId() == id) {
            user = tmp;
            break;
          }
        }

        if (user == null) {
          pair = new ArrayList();
          pair.add(new Pair(movieId, ratings));
          user = new User(id, pair, msdSimilarUsers);
          Pair tmp = (Pair)pair.get(0);
          userArrayList.add(user);



        } else
          user.getpairArray().add(new Pair(movieId, ratings));
          index++;
        }
        System.out.println("User arraylist contains users " +userArrayList.size());
        addTrainingData(userArrayList);
        System.out.println("User arraylist part2 contains users " +userArrayList.size());


    }

//*********************************************************************************************
    //A method for calculating the mean squared difference algorithm
//*********************************************************************************************
    public static float meanSquaredDifference(User user1, User user2) {
      float num = 0;
      float sum_diffs = 0;
      Iterator iter = user1.getpairArray().iterator();
      while (iter.hasNext()) {
        Pair pair1 = (Pair)iter.next();
        Iterator iter2 = user2.getpairArray().iterator();
        while (iter2.hasNext()) {
          Pair pair2 = (Pair)iter2.next();
          if (pair1.getId() == pair2.getId()) {
            float diff = pair1.getRatings() - pair2.getRatings();
            sum_diffs += diff*diff;
            num++;
          }

        }
      }

      if (num == 0)
        return 0;
      else
        return sum_diffs/num;
    }




    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }

       }
       return prediction;
   }
//*********************************************************************************************
    //Populate arraylists with the training data and randomly extract 20%
//*********************************************************************************************
    public static void addTrainingData(ArrayList userArrayList) {
      ArrayList fullArrayList = new ArrayList();
      ArrayList halfArrayList = new ArrayList();
      ArrayList similarUsers = new ArrayList();
      System.out.println("Useer al is" +userArrayList.size());
      System.out.println("It check");
      Iterator iter = userArrayList.iterator();
      System.out.println("U al is" +userArrayList.size());
      User userToCopy = null;
      User userToTest = null;
      int count = 0;
      ArrayList pairsToCopy = null;
      Pair curPair = null;
      Pair pairToTest = null;
      float actualRating = 0;
      float absoluteError = 0;
      float meanAbsoluteError = 0;
      //float predictedRating = 0;
      while (iter.hasNext()) {
        System.out.println("In here");
        userToCopy = (User) iter.next();
        pairsToCopy = new ArrayList();
        Iterator pairIt = userToCopy.getpairArray().iterator();
        while (pairIt.hasNext()) {
          curPair = (Pair) pairIt.next();
          pairsToCopy.add(new Pair(curPair.getId(), curPair.getRatings()));
        }
        Collections.shuffle(pairsToCopy);
        fullArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(0,
            ((pairsToCopy.size()/5)*4))),userToCopy.getMsdSimilarUsers()));
        System.out.println("Fullal Size is " +fullArrayList.size());
        halfArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(
            ((pairsToCopy.size() /5)*4), pairsToCopy.size())), userToCopy.getMsdSimilarUsers()));
        System.out.println("Size is " +halfArrayList.size());
      }
      Iterator iter3 = halfArrayList.iterator();
      while (iter3.hasNext()) {
        userToTest = (User)iter3.next();
        Iterator iter4 = userToTest.getpairArray().iterator();
        while (iter4.hasNext()) {
          pairToTest = (Pair) iter4.next();
          actualRating = pairToTest.getRatings();
          float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);
          absoluteError += (actualRating-predictedRating);
          count++;
          meanAbsoluteError = (absoluteError/count);
          System.out.println(userToTest.getUserId());

        }
      }
    }

}
//********************************************************************************************************************
   //The user class
//********************************************************************************************************************
class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;


    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;

    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }



}

//********************************************************************************************************************
   //The pair class
//********************************************************************************************************************
class Pair {

      int id; //Value to store id
      float ratings; //Value to store float

      public Pair(int id, float ratings) {
        this.id = id;
        this.ratings = ratings;
      }

      public int getId() {
        return id;
      }

      public float getRatings() {
        return ratings;
      }
}


And heres the portion of the dataset that i extracted to work with:

1      1      0.90
119      1      0.70
120        1          0.80
162      1           0.20
220       1           0.20

Thx.

O2K

In your main method, you only create one msdSimilarUsers ArrayList.  This gets added to each user in the line:

 user = new User(id, pair, msdSimilarUsers);

This may explain why changing one user's msdSimilarUsers data changes all of them (they're all referring to the same ArrayList).

Could you suggest a way in which to fix this poblem.I've tried setting up new arraylists and transferring values, but i keep getting the same result

Yeah what should happen is that the user should only be added to a users arraylist of similar users bif the mean squared difference between the users is less than 0.15. So msdSimilarUsers is only created after the whole file is read in, as only then are user profiles setup.We then need to calculate who the most similar users are and then add them into a users profile

Oh sorry read that last comment wrong :)

You're a star jimmack.Thx a million again for the solution.You've been a great help.Much appreciated!

;-)

Is there any particular reason why you're not asking questions in the Java area?

I didnt even realise that i wasn't posting in the java area!

I actually encountered another slight problem with my system.

basically when i call the generateMsdprediction method it returns me the last value calculated for prediction(as the method processes every user).

However what i want it to do is to return the value for prediction that is calculated on each iteration of the loop.Obviously getting the same value for prediction every time completely changes the functionality.

Is there anywy in which i could do this?

If I've understood you correctly, you actually want the results for each user returned.  If this is the case, you could do:

//    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+    public static ArrayList generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+     ArrayList results = new ArrayList();
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }
+         results.add(new Float(prediction));
       }
//       return prediction;
   }

(This lines with a + at the front are additions)

The exact fucntionlity i want is that when i call this method in the addtraining method:

float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);

should return 1 prediction for 1 movie.

So it should start by predicting what user 119 gave to movie 1.Retun this value to the addTraing method.
Then I subtract this prediction from the actaul rating, to give an error value.

Then predict what user 119 gave to movie 2, return this value,

Then what user 120 gave to movie 1 etc.

(Note: *user 119 and 120 are in user 1s arraylist of simUsers*)

Add up all the error values and divide by the number of predictions to get a mean error value.

I hope thats clear.If not let me know

I'm a bit pressed for time at the moment.  You should probably ask this as another question (since you've accepted my answer to your original question, no-one else is likely to comment in this thread now).

Oh ok thx.Will do so in the morning.Have been working on a solution, but am just muddling up the whole system now.