Serious debugging problem urgent....

OK the problem thats happening is that my arraylist of similar users should be populated as it iterates through each user.

However it isn't doing this correctlly.basically i have a method that takes user1 and user2 as parameteres.If the value returned by the method is less than a threshold then user2 should be added to user1s arraylist of similar users.

Whats happening is that user2s arraylist of similarusers is being populated at the same time.This shouldnt happen at all.Only user1s arraylist should be populated throughout the iterations of the loop.

Any help would be greatly appreciated.

heres the code(excuse all the debugging statements!):

-------------------------------------------
process
-------------------------------------------
 public static double[][] process(ArrayList userArrayList) {
      ArrayList similarUsers = new ArrayList();
      Object[] users = userArrayList.toArray();
      double[][] users_diffs = new double[users.length][users.length];
      for (int i = 0; i < users.length-1; i++)
        for (int j = i+1; j < users.length; j++) {
          User user1 = (User)users[i];
          User user2 = (User)users[j];
          users_diffs[i][j] = meanSquaredDifference(user1,user2);
 
          if((meanSquaredDifference(user1, user2) < 0.15 )) {
           

            //System.out.println("MSD is "  +meanSquaredDifference(user1, user2));
            //System.out.println("User1s id is" +user1.getUserId());
            //System.out.println("User2s id is" +user2.getUserId());
            //System.out.println("Start size of user 1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Start size of user 2s AL is " +user2.msdSimilarUsers.size());
           

            user1.msdSimilarUsers.add(user2);

           

            //user2.msdSimilarUsers.remove(user1);
            //System.out.println("Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Updated Size of user2s AL is " +user2.msdSimilarUsers.size());
       


********************
I cant understan why the next line empties user1 and user2s arraylists
*********************

            //user2.msdSimilarUsers.remove(user2);


            //User tmp = (User)user2.msdSimilarUsers.get(0);
            //System.out.print("User added to 2s list is " +tmp.getUserId());
            //System.out.println("Next Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Size of AL is " +user2.msdSimilarUsers.size());

            //user2.msdSimilarUsers.add(user1);
            //System.out.println("User1 sim users size is" +user1.msdSimilarUsers.size());
          }
      }
    return users_diffs;
  }


_____________________________
Mean squared difference
--------------------------------------------
  public static float meanSquaredDifference(User user1, User user2) {
      float num = 0;
      float sum_diffs = 0;
      Iterator iter = user1.getpairArray().iterator();
      while (iter.hasNext()) {
        Pair pair1 = (Pair)iter.next();
        Iterator iter2 = user2.getpairArray().iterator();
        while (iter2.hasNext()) {
          Pair pair2 = (Pair)iter2.next();
          if (pair1.getId() == pair2.getId()) {
            float diff = pair1.getRatings() - pair2.getRatings();
            sum_diffs += diff*diff;
            num++;
          }

        }
      }

      if (num == 0)
        return 0;
      else
        return sum_diffs/num;
    }



This is an urgent problem, so please respond ASAP.I will be available all day Sat/Sun.

Regards.

O2K
LVL 2
override_y2kAsked:
Who is Participating?
 
jimmackCommented:
Since you don't seem to be adding any data to the msdSimilarUsers before you create the new User objects, you could replace:

         user = new User(id, pair, msdSimilarUsers);

with

         user = new User(id, pair, new ArrayList());

If I've missed something (and you do have data in it before creating a User), create a new msdSimilarUsers ArrayList inside the while loop:

      while((line = inFile.readLine()) != null) {

        msdSimilarUsers = new ArrayList();

        tokenizer = new StringTokenizer(line);     // tokenize the file line
        // now add the user id to the user id array
 
0
 
jimmackCommented:
You need to look earlier in your code, somewhere before you call the process() method.

If user1 and user2 are both being affected by a command (eg. add()), then they must both be referencing the same object, so:

          User user1 = (User)users[i];
          User user2 = (User)users[j];

must mean that the users array contains references to the same object.  You need to check how the users array is being created.
0
 
override_y2kAuthor Commented:
When i throw in statements to give me user1s id and user 2s id here, it gives me the correct ids:

          User user1 = (User)users[i];
          System.out.println("User1s id is" + user1.getUserId());
          User user2 = (User)users[j];
          System.out.println("User2s id is" + user2.getUserId());

Every iteration prints out the correct user id.

The users array is created like this:

Object[] users = userArrayList.toArray();

Is that causing a problem because i'm now referencing an array instead of an arraylist?
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

 
jimmackCommented:
Ah.  The problem is not with the users, it is with the msdSimilarUsers attribute within them.

Either you have declared this as static (so all users share the same msdSimilarUsers data) or when you set up the msdSimilarUsers attribute, you may be adding references to the same object to each of the User objects.

I'd check to see if this is static first ;-)
0
 
override_y2kAuthor Commented:
Here's the user class where i have the arraylist of sim users setup:

class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;

    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;
    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }

}
0
 
override_y2kAuthor Commented:
I do make a static declaration at the start of my program:

class ***

  private static ArrayList userArrayList = new ArrayList();

Would that be the problem?
0
 
override_y2kAuthor Commented:
I did some further debugging and what happens exactly on the 1st iteration is that user2 is added to user1s and user2s arraylist of similarUsers.

On the next iteration it adds another user into both arraylists.

Then it moves on from user1 and user1 now becomes user2 and user2 becomes3 etc as each step takes place.

However thats where the funny stuff starts to happen as it keeps populating the original users arrylist with similar users even though that user is no longer being used in the comparison!

I also tested each users arraylist of similar users and they all contain the exact same users(including replicas of certain users!)
0
 
override_y2kAuthor Commented:
Heres all the relevant code if anybody wants to have a look:

import java.io.*;
import java.util.*;

class FileParser {

  private static ArrayList userArrayList = new ArrayList();

  public static void main(String[] args) throws Exception {

    FileReader fr =
      new FileReader("C:/dataset1.txt"); // to open the file
      BufferedReader inFile = new BufferedReader(fr);     // to read the file
      String line;     // String variable used to hold each line of the file
      StringTokenizer tokenizer;     // string tokenizer to parse each file line
      ArrayList userArrayList = new ArrayList();
      ArrayList msdSimilarUsers = new ArrayList();
      ArrayList pearsonsSimilarUsers = new ArrayList();
      ArrayList fullArrayList = new ArrayList();
      int id = 0;
      int index = 0;     // Index that we increment each time through the loop
      //User user = null;
      int movieId;     // value to hold movie id
      float ratings;     // value to hold movie ratings
      ArrayList pair = null;
      while((line = inFile.readLine()) != null) {
        tokenizer = new StringTokenizer(line);     // tokenize the file line
        // now add the user id to the user id array
        id = Integer.parseInt(tokenizer.nextToken());
        // initialize the movie id variable
        movieId = Integer.parseInt(tokenizer.nextToken());
        // initialize the cd ratings variable
        ratings = Float.parseFloat(tokenizer.nextToken());
        // Check out if the user has already been added to the list
        Iterator iter = userArrayList.iterator();
        User user = null;
        while (iter.hasNext()) {
          User tmp = (User)iter.next();
          if (tmp.getUserId() == id) {
            user = tmp;
            break;
          }
        }

        if (user == null) {
          pair = new ArrayList();
          pair.add(new Pair(movieId, ratings));
          user = new User(id, pair, msdSimilarUsers);
          Pair tmp = (Pair)pair.get(0);
          userArrayList.add(user);



        } else
          user.getpairArray().add(new Pair(movieId, ratings));
          index++;
        }
        System.out.println("User arraylist contains users " +userArrayList.size());
        addTrainingData(userArrayList);
        System.out.println("User arraylist part2 contains users " +userArrayList.size());


    }

//*********************************************************************************************
    //A method for calculating the mean squared difference algorithm
//*********************************************************************************************
    public static float meanSquaredDifference(User user1, User user2) {
      float num = 0;
      float sum_diffs = 0;
      Iterator iter = user1.getpairArray().iterator();
      while (iter.hasNext()) {
        Pair pair1 = (Pair)iter.next();
        Iterator iter2 = user2.getpairArray().iterator();
        while (iter2.hasNext()) {
          Pair pair2 = (Pair)iter2.next();
          if (pair1.getId() == pair2.getId()) {
            float diff = pair1.getRatings() - pair2.getRatings();
            sum_diffs += diff*diff;
            num++;
          }

        }
      }

      if (num == 0)
        return 0;
      else
        return sum_diffs/num;
    }




    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }

       }
       return prediction;
   }
//*********************************************************************************************
    //Populate arraylists with the training data and randomly extract 20%
//*********************************************************************************************
    public static void addTrainingData(ArrayList userArrayList) {
      ArrayList fullArrayList = new ArrayList();
      ArrayList halfArrayList = new ArrayList();
      ArrayList similarUsers = new ArrayList();
      System.out.println("Useer al is" +userArrayList.size());
      System.out.println("It check");
      Iterator iter = userArrayList.iterator();
      System.out.println("U al is" +userArrayList.size());
      User userToCopy = null;
      User userToTest = null;
      int count = 0;
      ArrayList pairsToCopy = null;
      Pair curPair = null;
      Pair pairToTest = null;
      float actualRating = 0;
      float absoluteError = 0;
      float meanAbsoluteError = 0;
      //float predictedRating = 0;
      while (iter.hasNext()) {
        System.out.println("In here");
        userToCopy = (User) iter.next();
        pairsToCopy = new ArrayList();
        Iterator pairIt = userToCopy.getpairArray().iterator();
        while (pairIt.hasNext()) {
          curPair = (Pair) pairIt.next();
          pairsToCopy.add(new Pair(curPair.getId(), curPair.getRatings()));
        }
        Collections.shuffle(pairsToCopy);
        fullArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(0,
            ((pairsToCopy.size()/5)*4))),userToCopy.getMsdSimilarUsers()));
        System.out.println("Fullal Size is " +fullArrayList.size());
        halfArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(
            ((pairsToCopy.size() /5)*4), pairsToCopy.size())), userToCopy.getMsdSimilarUsers()));
        System.out.println("Size is " +halfArrayList.size());
      }
      Iterator iter3 = halfArrayList.iterator();
      while (iter3.hasNext()) {
        userToTest = (User)iter3.next();
        Iterator iter4 = userToTest.getpairArray().iterator();
        while (iter4.hasNext()) {
          pairToTest = (Pair) iter4.next();
          actualRating = pairToTest.getRatings();
          float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);
          absoluteError += (actualRating-predictedRating);
          count++;
          meanAbsoluteError = (absoluteError/count);
          System.out.println(userToTest.getUserId());

        }
      }
    }

}
//********************************************************************************************************************
   //The user class
//********************************************************************************************************************
class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;


    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;

    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }



}

//********************************************************************************************************************
   //The pair class
//********************************************************************************************************************
class Pair {

      int id; //Value to store id
      float ratings; //Value to store float

      public Pair(int id, float ratings) {
        this.id = id;
        this.ratings = ratings;
      }

      public int getId() {
        return id;
      }

      public float getRatings() {
        return ratings;
      }
}


And heres the portion of the dataset that i extracted to work with:

1      1      0.90
119      1      0.70
120        1          0.80
162      1           0.20
220       1           0.20

Thx.

O2K
0
 
jimmackCommented:
In your main method, you only create one msdSimilarUsers ArrayList.  This gets added to each user in the line:

 user = new User(id, pair, msdSimilarUsers);

This may explain why changing one user's msdSimilarUsers data changes all of them (they're all referring to the same ArrayList).
0
 
override_y2kAuthor Commented:
Could you suggest a way in which to fix this poblem.I've tried setting up new arraylists and transferring values, but i keep getting the same result
0
 
override_y2kAuthor Commented:
Yeah what should happen is that the user should only be added to a users arraylist of similar users bif the mean squared difference between the users is less than 0.15. So msdSimilarUsers is only created after the whole file is read in, as only then are user profiles setup.We then need to calculate who the most similar users are and then add them into a users profile
0
 
override_y2kAuthor Commented:
Oh sorry read that last comment wrong :)
0
 
override_y2kAuthor Commented:
You're a star jimmack.Thx a million again for the solution.You've been a great help.Much appreciated!
0
 
jimmackCommented:
;-)
0
 
jimmackCommented:
Is there any particular reason why you're not asking questions in the Java area?
0
 
override_y2kAuthor Commented:
I didnt even realise that i wasn't posting in the java area!

I actually encountered another slight problem with my system.

basically when i call the generateMsdprediction method it returns me the last value calculated for prediction(as the method processes every user).

However what i want it to do is to return the value for prediction that is calculated on each iteration of the loop.Obviously getting the same value for prediction every time completely changes the functionality.

Is there anywy in which i could do this?
0
 
jimmackCommented:
If I've understood you correctly, you actually want the results for each user returned.  If this is the case, you could do:

//    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+    public static ArrayList generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+     ArrayList results = new ArrayList();
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }
+         results.add(new Float(prediction));
       }
//       return prediction;
   }

(This lines with a + at the front are additions)
0
 
override_y2kAuthor Commented:
The exact fucntionlity i want is that when i call this method in the addtraining method:

float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);

should return 1 prediction for 1 movie.

So it should start by predicting what user 119 gave to movie 1.Retun this value to the addTraing method.
Then I subtract this prediction from the actaul rating, to give an error value.

Then predict what user 119 gave to movie 2, return this value,

Then what user 120 gave to movie 1 etc.

(Note: *user 119 and 120 are in user 1s arraylist of simUsers*)

Add up all the error values and divide by the number of predictions to get a mean error value.

I hope thats clear.If not let me know

0
 
jimmackCommented:
I'm a bit pressed for time at the moment.  You should probably ask this as another question (since you've accepted my answer to your original question, no-one else is likely to comment in this thread now).
0
 
override_y2kAuthor Commented:
Oh ok thx.Will do so in the morning.Have been working on a solution, but am just muddling up the whole system now.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.