Serious debugging problem urgent....

OK the problem thats happening is that my arraylist of similar users should be populated as it iterates through each user.

However it isn't doing this correctlly.basically i have a method that takes user1 and user2 as parameteres.If the value returned by the method is less than a threshold then user2 should be added to user1s arraylist of similar users.

Whats happening is that user2s arraylist of similarusers is being populated at the same time.This shouldnt happen at all.Only user1s arraylist should be populated throughout the iterations of the loop.

Any help would be greatly appreciated.

heres the code(excuse all the debugging statements!):

-------------------------------------------
process
-------------------------------------------
 public static double[][] process(ArrayList userArrayList) {
      ArrayList similarUsers = new ArrayList();
      Object[] users = userArrayList.toArray();
      double[][] users_diffs = new double[users.length][users.length];
      for (int i = 0; i < users.length-1; i++)
        for (int j = i+1; j < users.length; j++) {
          User user1 = (User)users[i];
          User user2 = (User)users[j];
          users_diffs[i][j] = meanSquaredDifference(user1,user2);
 
          if((meanSquaredDifference(user1, user2) < 0.15 )) {
           

            //System.out.println("MSD is "  +meanSquaredDifference(user1, user2));
            //System.out.println("User1s id is" +user1.getUserId());
            //System.out.println("User2s id is" +user2.getUserId());
            //System.out.println("Start size of user 1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Start size of user 2s AL is " +user2.msdSimilarUsers.size());
           

            user1.msdSimilarUsers.add(user2);

           

            //user2.msdSimilarUsers.remove(user1);
            //System.out.println("Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Updated Size of user2s AL is " +user2.msdSimilarUsers.size());
       


********************
I cant understan why the next line empties user1 and user2s arraylists
*********************

            //user2.msdSimilarUsers.remove(user2);


            //User tmp = (User)user2.msdSimilarUsers.get(0);
            //System.out.print("User added to 2s list is " +tmp.getUserId());
            //System.out.println("Next Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Size of AL is " +user2.msdSimilarUsers.size());

            //user2.msdSimilarUsers.add(user1);
            //System.out.println("User1 sim users size is" +user1.msdSimilarUsers.size());
          }
      }
    return users_diffs;
  }


_____________________________
Mean squared difference
--------------------------------------------
  public static float meanSquaredDifference(User user1, User user2) {
      float num = 0;
      float sum_diffs = 0;
      Iterator iter = user1.getpairArray().iterator();
      while (iter.hasNext()) {
        Pair pair1 = (Pair)iter.next();
        Iterator iter2 = user2.getpairArray().iterator();
        while (iter2.hasNext()) {
          Pair pair2 = (Pair)iter2.next();
          if (pair1.getId() == pair2.getId()) {
            float diff = pair1.getRatings() - pair2.getRatings();
            sum_diffs += diff*diff;
            num++;
          }

        }
      }

      if (num == 0)
        return 0;
      else
        return sum_diffs/num;
    }



This is an urgent problem, so please respond ASAP.I will be available all day Sat/Sun.

Regards.

O2K
LVL 2
override_y2kAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jimmackCommented:
You need to look earlier in your code, somewhere before you call the process() method.

If user1 and user2 are both being affected by a command (eg. add()), then they must both be referencing the same object, so:

          User user1 = (User)users[i];
          User user2 = (User)users[j];

must mean that the users array contains references to the same object.  You need to check how the users array is being created.
override_y2kAuthor Commented:
When i throw in statements to give me user1s id and user 2s id here, it gives me the correct ids:

          User user1 = (User)users[i];
          System.out.println("User1s id is" + user1.getUserId());
          User user2 = (User)users[j];
          System.out.println("User2s id is" + user2.getUserId());

Every iteration prints out the correct user id.

The users array is created like this:

Object[] users = userArrayList.toArray();

Is that causing a problem because i'm now referencing an array instead of an arraylist?
jimmackCommented:
Ah.  The problem is not with the users, it is with the msdSimilarUsers attribute within them.

Either you have declared this as static (so all users share the same msdSimilarUsers data) or when you set up the msdSimilarUsers attribute, you may be adding references to the same object to each of the User objects.

I'd check to see if this is static first ;-)
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

override_y2kAuthor Commented:
Here's the user class where i have the arraylist of sim users setup:

class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;

    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;
    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }

}
override_y2kAuthor Commented:
I do make a static declaration at the start of my program:

class ***

  private static ArrayList userArrayList = new ArrayList();

Would that be the problem?
override_y2kAuthor Commented:
I did some further debugging and what happens exactly on the 1st iteration is that user2 is added to user1s and user2s arraylist of similarUsers.

On the next iteration it adds another user into both arraylists.

Then it moves on from user1 and user1 now becomes user2 and user2 becomes3 etc as each step takes place.

However thats where the funny stuff starts to happen as it keeps populating the original users arrylist with similar users even though that user is no longer being used in the comparison!

I also tested each users arraylist of similar users and they all contain the exact same users(including replicas of certain users!)
override_y2kAuthor Commented:
Heres all the relevant code if anybody wants to have a look:

import java.io.*;
import java.util.*;

class FileParser {

  private static ArrayList userArrayList = new ArrayList();

  public static void main(String[] args) throws Exception {

    FileReader fr =
      new FileReader("C:/dataset1.txt"); // to open the file
      BufferedReader inFile = new BufferedReader(fr);     // to read the file
      String line;     // String variable used to hold each line of the file
      StringTokenizer tokenizer;     // string tokenizer to parse each file line
      ArrayList userArrayList = new ArrayList();
      ArrayList msdSimilarUsers = new ArrayList();
      ArrayList pearsonsSimilarUsers = new ArrayList();
      ArrayList fullArrayList = new ArrayList();
      int id = 0;
      int index = 0;     // Index that we increment each time through the loop
      //User user = null;
      int movieId;     // value to hold movie id
      float ratings;     // value to hold movie ratings
      ArrayList pair = null;
      while((line = inFile.readLine()) != null) {
        tokenizer = new StringTokenizer(line);     // tokenize the file line
        // now add the user id to the user id array
        id = Integer.parseInt(tokenizer.nextToken());
        // initialize the movie id variable
        movieId = Integer.parseInt(tokenizer.nextToken());
        // initialize the cd ratings variable
        ratings = Float.parseFloat(tokenizer.nextToken());
        // Check out if the user has already been added to the list
        Iterator iter = userArrayList.iterator();
        User user = null;
        while (iter.hasNext()) {
          User tmp = (User)iter.next();
          if (tmp.getUserId() == id) {
            user = tmp;
            break;
          }
        }

        if (user == null) {
          pair = new ArrayList();
          pair.add(new Pair(movieId, ratings));
          user = new User(id, pair, msdSimilarUsers);
          Pair tmp = (Pair)pair.get(0);
          userArrayList.add(user);



        } else
          user.getpairArray().add(new Pair(movieId, ratings));
          index++;
        }
        System.out.println("User arraylist contains users " +userArrayList.size());
        addTrainingData(userArrayList);
        System.out.println("User arraylist part2 contains users " +userArrayList.size());


    }

//*********************************************************************************************
    //A method for calculating the mean squared difference algorithm
//*********************************************************************************************
    public static float meanSquaredDifference(User user1, User user2) {
      float num = 0;
      float sum_diffs = 0;
      Iterator iter = user1.getpairArray().iterator();
      while (iter.hasNext()) {
        Pair pair1 = (Pair)iter.next();
        Iterator iter2 = user2.getpairArray().iterator();
        while (iter2.hasNext()) {
          Pair pair2 = (Pair)iter2.next();
          if (pair1.getId() == pair2.getId()) {
            float diff = pair1.getRatings() - pair2.getRatings();
            sum_diffs += diff*diff;
            num++;
          }

        }
      }

      if (num == 0)
        return 0;
      else
        return sum_diffs/num;
    }




    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }

       }
       return prediction;
   }
//*********************************************************************************************
    //Populate arraylists with the training data and randomly extract 20%
//*********************************************************************************************
    public static void addTrainingData(ArrayList userArrayList) {
      ArrayList fullArrayList = new ArrayList();
      ArrayList halfArrayList = new ArrayList();
      ArrayList similarUsers = new ArrayList();
      System.out.println("Useer al is" +userArrayList.size());
      System.out.println("It check");
      Iterator iter = userArrayList.iterator();
      System.out.println("U al is" +userArrayList.size());
      User userToCopy = null;
      User userToTest = null;
      int count = 0;
      ArrayList pairsToCopy = null;
      Pair curPair = null;
      Pair pairToTest = null;
      float actualRating = 0;
      float absoluteError = 0;
      float meanAbsoluteError = 0;
      //float predictedRating = 0;
      while (iter.hasNext()) {
        System.out.println("In here");
        userToCopy = (User) iter.next();
        pairsToCopy = new ArrayList();
        Iterator pairIt = userToCopy.getpairArray().iterator();
        while (pairIt.hasNext()) {
          curPair = (Pair) pairIt.next();
          pairsToCopy.add(new Pair(curPair.getId(), curPair.getRatings()));
        }
        Collections.shuffle(pairsToCopy);
        fullArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(0,
            ((pairsToCopy.size()/5)*4))),userToCopy.getMsdSimilarUsers()));
        System.out.println("Fullal Size is " +fullArrayList.size());
        halfArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(
            ((pairsToCopy.size() /5)*4), pairsToCopy.size())), userToCopy.getMsdSimilarUsers()));
        System.out.println("Size is " +halfArrayList.size());
      }
      Iterator iter3 = halfArrayList.iterator();
      while (iter3.hasNext()) {
        userToTest = (User)iter3.next();
        Iterator iter4 = userToTest.getpairArray().iterator();
        while (iter4.hasNext()) {
          pairToTest = (Pair) iter4.next();
          actualRating = pairToTest.getRatings();
          float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);
          absoluteError += (actualRating-predictedRating);
          count++;
          meanAbsoluteError = (absoluteError/count);
          System.out.println(userToTest.getUserId());

        }
      }
    }

}
//********************************************************************************************************************
   //The user class
//********************************************************************************************************************
class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;


    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;

    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }



}

//********************************************************************************************************************
   //The pair class
//********************************************************************************************************************
class Pair {

      int id; //Value to store id
      float ratings; //Value to store float

      public Pair(int id, float ratings) {
        this.id = id;
        this.ratings = ratings;
      }

      public int getId() {
        return id;
      }

      public float getRatings() {
        return ratings;
      }
}


And heres the portion of the dataset that i extracted to work with:

1      1      0.90
119      1      0.70
120        1          0.80
162      1           0.20
220       1           0.20

Thx.

O2K
jimmackCommented:
In your main method, you only create one msdSimilarUsers ArrayList.  This gets added to each user in the line:

 user = new User(id, pair, msdSimilarUsers);

This may explain why changing one user's msdSimilarUsers data changes all of them (they're all referring to the same ArrayList).
override_y2kAuthor Commented:
Could you suggest a way in which to fix this poblem.I've tried setting up new arraylists and transferring values, but i keep getting the same result
jimmackCommented:
Since you don't seem to be adding any data to the msdSimilarUsers before you create the new User objects, you could replace:

         user = new User(id, pair, msdSimilarUsers);

with

         user = new User(id, pair, new ArrayList());

If I've missed something (and you do have data in it before creating a User), create a new msdSimilarUsers ArrayList inside the while loop:

      while((line = inFile.readLine()) != null) {

        msdSimilarUsers = new ArrayList();

        tokenizer = new StringTokenizer(line);     // tokenize the file line
        // now add the user id to the user id array
 

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
override_y2kAuthor Commented:
Yeah what should happen is that the user should only be added to a users arraylist of similar users bif the mean squared difference between the users is less than 0.15. So msdSimilarUsers is only created after the whole file is read in, as only then are user profiles setup.We then need to calculate who the most similar users are and then add them into a users profile
override_y2kAuthor Commented:
Oh sorry read that last comment wrong :)
override_y2kAuthor Commented:
You're a star jimmack.Thx a million again for the solution.You've been a great help.Much appreciated!
jimmackCommented:
;-)
jimmackCommented:
Is there any particular reason why you're not asking questions in the Java area?
override_y2kAuthor Commented:
I didnt even realise that i wasn't posting in the java area!

I actually encountered another slight problem with my system.

basically when i call the generateMsdprediction method it returns me the last value calculated for prediction(as the method processes every user).

However what i want it to do is to return the value for prediction that is calculated on each iteration of the loop.Obviously getting the same value for prediction every time completely changes the functionality.

Is there anywy in which i could do this?
jimmackCommented:
If I've understood you correctly, you actually want the results for each user returned.  If this is the case, you could do:

//    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+    public static ArrayList generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+     ArrayList results = new ArrayList();
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }
+         results.add(new Float(prediction));
       }
//       return prediction;
   }

(This lines with a + at the front are additions)
override_y2kAuthor Commented:
The exact fucntionlity i want is that when i call this method in the addtraining method:

float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);

should return 1 prediction for 1 movie.

So it should start by predicting what user 119 gave to movie 1.Retun this value to the addTraing method.
Then I subtract this prediction from the actaul rating, to give an error value.

Then predict what user 119 gave to movie 2, return this value,

Then what user 120 gave to movie 1 etc.

(Note: *user 119 and 120 are in user 1s arraylist of simUsers*)

Add up all the error values and divide by the number of predictions to get a mean error value.

I hope thats clear.If not let me know

jimmackCommented:
I'm a bit pressed for time at the moment.  You should probably ask this as another question (since you've accepted my answer to your original question, no-one else is likely to comment in this thread now).
override_y2kAuthor Commented:
Oh ok thx.Will do so in the morning.Have been working on a solution, but am just muddling up the whole system now.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Programming

From novice to tech pro — start learning today.