Solved

Serious debugging problem urgent....

Posted on 2003-12-06
20
314 Views
Last Modified: 2010-04-17
OK the problem thats happening is that my arraylist of similar users should be populated as it iterates through each user.

However it isn't doing this correctlly.basically i have a method that takes user1 and user2 as parameteres.If the value returned by the method is less than a threshold then user2 should be added to user1s arraylist of similar users.

Whats happening is that user2s arraylist of similarusers is being populated at the same time.This shouldnt happen at all.Only user1s arraylist should be populated throughout the iterations of the loop.

Any help would be greatly appreciated.

heres the code(excuse all the debugging statements!):

-------------------------------------------
process
-------------------------------------------
 public static double[][] process(ArrayList userArrayList) {
      ArrayList similarUsers = new ArrayList();
      Object[] users = userArrayList.toArray();
      double[][] users_diffs = new double[users.length][users.length];
      for (int i = 0; i < users.length-1; i++)
        for (int j = i+1; j < users.length; j++) {
          User user1 = (User)users[i];
          User user2 = (User)users[j];
          users_diffs[i][j] = meanSquaredDifference(user1,user2);
 
          if((meanSquaredDifference(user1, user2) < 0.15 )) {
           

            //System.out.println("MSD is "  +meanSquaredDifference(user1, user2));
            //System.out.println("User1s id is" +user1.getUserId());
            //System.out.println("User2s id is" +user2.getUserId());
            //System.out.println("Start size of user 1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Start size of user 2s AL is " +user2.msdSimilarUsers.size());
           

            user1.msdSimilarUsers.add(user2);

           

            //user2.msdSimilarUsers.remove(user1);
            //System.out.println("Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Updated Size of user2s AL is " +user2.msdSimilarUsers.size());
       


********************
I cant understan why the next line empties user1 and user2s arraylists
*********************

            //user2.msdSimilarUsers.remove(user2);


            //User tmp = (User)user2.msdSimilarUsers.get(0);
            //System.out.print("User added to 2s list is " +tmp.getUserId());
            //System.out.println("Next Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Updated Size of user1s AL is " +user1.msdSimilarUsers.size());
            //System.out.println("Size of AL is " +user2.msdSimilarUsers.size());

            //user2.msdSimilarUsers.add(user1);
            //System.out.println("User1 sim users size is" +user1.msdSimilarUsers.size());
          }
      }
    return users_diffs;
  }


_____________________________
Mean squared difference
--------------------------------------------
  public static float meanSquaredDifference(User user1, User user2) {
      float num = 0;
      float sum_diffs = 0;
      Iterator iter = user1.getpairArray().iterator();
      while (iter.hasNext()) {
        Pair pair1 = (Pair)iter.next();
        Iterator iter2 = user2.getpairArray().iterator();
        while (iter2.hasNext()) {
          Pair pair2 = (Pair)iter2.next();
          if (pair1.getId() == pair2.getId()) {
            float diff = pair1.getRatings() - pair2.getRatings();
            sum_diffs += diff*diff;
            num++;
          }

        }
      }

      if (num == 0)
        return 0;
      else
        return sum_diffs/num;
    }



This is an urgent problem, so please respond ASAP.I will be available all day Sat/Sun.

Regards.

O2K
0
Comment
Question by:override_y2k
  • 12
  • 8
20 Comments
 
LVL 15

Expert Comment

by:jimmack
ID: 9891886
You need to look earlier in your code, somewhere before you call the process() method.

If user1 and user2 are both being affected by a command (eg. add()), then they must both be referencing the same object, so:

          User user1 = (User)users[i];
          User user2 = (User)users[j];

must mean that the users array contains references to the same object.  You need to check how the users array is being created.
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9892153
When i throw in statements to give me user1s id and user 2s id here, it gives me the correct ids:

          User user1 = (User)users[i];
          System.out.println("User1s id is" + user1.getUserId());
          User user2 = (User)users[j];
          System.out.println("User2s id is" + user2.getUserId());

Every iteration prints out the correct user id.

The users array is created like this:

Object[] users = userArrayList.toArray();

Is that causing a problem because i'm now referencing an array instead of an arraylist?
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9892193
Ah.  The problem is not with the users, it is with the msdSimilarUsers attribute within them.

Either you have declared this as static (so all users share the same msdSimilarUsers data) or when you set up the msdSimilarUsers attribute, you may be adding references to the same object to each of the User objects.

I'd check to see if this is static first ;-)
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9892501
Here's the user class where i have the arraylist of sim users setup:

class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;

    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;
    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }

}
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9892502
I do make a static declaration at the start of my program:

class ***

  private static ArrayList userArrayList = new ArrayList();

Would that be the problem?
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9892596
I did some further debugging and what happens exactly on the 1st iteration is that user2 is added to user1s and user2s arraylist of similarUsers.

On the next iteration it adds another user into both arraylists.

Then it moves on from user1 and user1 now becomes user2 and user2 becomes3 etc as each step takes place.

However thats where the funny stuff starts to happen as it keeps populating the original users arrylist with similar users even though that user is no longer being used in the comparison!

I also tested each users arraylist of similar users and they all contain the exact same users(including replicas of certain users!)
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9892647
Heres all the relevant code if anybody wants to have a look:

import java.io.*;
import java.util.*;

class FileParser {

  private static ArrayList userArrayList = new ArrayList();

  public static void main(String[] args) throws Exception {

    FileReader fr =
      new FileReader("C:/dataset1.txt"); // to open the file
      BufferedReader inFile = new BufferedReader(fr);     // to read the file
      String line;     // String variable used to hold each line of the file
      StringTokenizer tokenizer;     // string tokenizer to parse each file line
      ArrayList userArrayList = new ArrayList();
      ArrayList msdSimilarUsers = new ArrayList();
      ArrayList pearsonsSimilarUsers = new ArrayList();
      ArrayList fullArrayList = new ArrayList();
      int id = 0;
      int index = 0;     // Index that we increment each time through the loop
      //User user = null;
      int movieId;     // value to hold movie id
      float ratings;     // value to hold movie ratings
      ArrayList pair = null;
      while((line = inFile.readLine()) != null) {
        tokenizer = new StringTokenizer(line);     // tokenize the file line
        // now add the user id to the user id array
        id = Integer.parseInt(tokenizer.nextToken());
        // initialize the movie id variable
        movieId = Integer.parseInt(tokenizer.nextToken());
        // initialize the cd ratings variable
        ratings = Float.parseFloat(tokenizer.nextToken());
        // Check out if the user has already been added to the list
        Iterator iter = userArrayList.iterator();
        User user = null;
        while (iter.hasNext()) {
          User tmp = (User)iter.next();
          if (tmp.getUserId() == id) {
            user = tmp;
            break;
          }
        }

        if (user == null) {
          pair = new ArrayList();
          pair.add(new Pair(movieId, ratings));
          user = new User(id, pair, msdSimilarUsers);
          Pair tmp = (Pair)pair.get(0);
          userArrayList.add(user);



        } else
          user.getpairArray().add(new Pair(movieId, ratings));
          index++;
        }
        System.out.println("User arraylist contains users " +userArrayList.size());
        addTrainingData(userArrayList);
        System.out.println("User arraylist part2 contains users " +userArrayList.size());


    }

//*********************************************************************************************
    //A method for calculating the mean squared difference algorithm
//*********************************************************************************************
    public static float meanSquaredDifference(User user1, User user2) {
      float num = 0;
      float sum_diffs = 0;
      Iterator iter = user1.getpairArray().iterator();
      while (iter.hasNext()) {
        Pair pair1 = (Pair)iter.next();
        Iterator iter2 = user2.getpairArray().iterator();
        while (iter2.hasNext()) {
          Pair pair2 = (Pair)iter2.next();
          if (pair1.getId() == pair2.getId()) {
            float diff = pair1.getRatings() - pair2.getRatings();
            sum_diffs += diff*diff;
            num++;
          }

        }
      }

      if (num == 0)
        return 0;
      else
        return sum_diffs/num;
    }




    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }

       }
       return prediction;
   }
//*********************************************************************************************
    //Populate arraylists with the training data and randomly extract 20%
//*********************************************************************************************
    public static void addTrainingData(ArrayList userArrayList) {
      ArrayList fullArrayList = new ArrayList();
      ArrayList halfArrayList = new ArrayList();
      ArrayList similarUsers = new ArrayList();
      System.out.println("Useer al is" +userArrayList.size());
      System.out.println("It check");
      Iterator iter = userArrayList.iterator();
      System.out.println("U al is" +userArrayList.size());
      User userToCopy = null;
      User userToTest = null;
      int count = 0;
      ArrayList pairsToCopy = null;
      Pair curPair = null;
      Pair pairToTest = null;
      float actualRating = 0;
      float absoluteError = 0;
      float meanAbsoluteError = 0;
      //float predictedRating = 0;
      while (iter.hasNext()) {
        System.out.println("In here");
        userToCopy = (User) iter.next();
        pairsToCopy = new ArrayList();
        Iterator pairIt = userToCopy.getpairArray().iterator();
        while (pairIt.hasNext()) {
          curPair = (Pair) pairIt.next();
          pairsToCopy.add(new Pair(curPair.getId(), curPair.getRatings()));
        }
        Collections.shuffle(pairsToCopy);
        fullArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(0,
            ((pairsToCopy.size()/5)*4))),userToCopy.getMsdSimilarUsers()));
        System.out.println("Fullal Size is " +fullArrayList.size());
        halfArrayList.add(new User(userToCopy.getUserId(),
                                   new ArrayList(pairsToCopy.subList(
            ((pairsToCopy.size() /5)*4), pairsToCopy.size())), userToCopy.getMsdSimilarUsers()));
        System.out.println("Size is " +halfArrayList.size());
      }
      Iterator iter3 = halfArrayList.iterator();
      while (iter3.hasNext()) {
        userToTest = (User)iter3.next();
        Iterator iter4 = userToTest.getpairArray().iterator();
        while (iter4.hasNext()) {
          pairToTest = (Pair) iter4.next();
          actualRating = pairToTest.getRatings();
          float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);
          absoluteError += (actualRating-predictedRating);
          count++;
          meanAbsoluteError = (absoluteError/count);
          System.out.println(userToTest.getUserId());

        }
      }
    }

}
//********************************************************************************************************************
   //The user class
//********************************************************************************************************************
class User {

    int userId; //Value to store id
    ArrayList pairArray; //Value to store float
    ArrayList msdSimilarUsers;


    public User(int userId, ArrayList pairArray, ArrayList msdSimilarUsers) {
      this.userId = userId;
      this.pairArray = pairArray;
      this.msdSimilarUsers = msdSimilarUsers;

    }

    public int getUserId() {
      return userId;
    }

    public ArrayList getpairArray() {
      return pairArray;
    }

    public ArrayList getMsdSimilarUsers() {
      return msdSimilarUsers;
    }



}

//********************************************************************************************************************
   //The pair class
//********************************************************************************************************************
class Pair {

      int id; //Value to store id
      float ratings; //Value to store float

      public Pair(int id, float ratings) {
        this.id = id;
        this.ratings = ratings;
      }

      public int getId() {
        return id;
      }

      public float getRatings() {
        return ratings;
      }
}


And heres the portion of the dataset that i extracted to work with:

1      1      0.90
119      1      0.70
120        1          0.80
162      1           0.20
220       1           0.20

Thx.

O2K
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9892697
In your main method, you only create one msdSimilarUsers ArrayList.  This gets added to each user in the line:

 user = new User(id, pair, msdSimilarUsers);

This may explain why changing one user's msdSimilarUsers data changes all of them (they're all referring to the same ArrayList).
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9893132
Could you suggest a way in which to fix this poblem.I've tried setting up new arraylists and transferring values, but i keep getting the same result
0
 
LVL 15

Accepted Solution

by:
jimmack earned 200 total points
ID: 9893366
Since you don't seem to be adding any data to the msdSimilarUsers before you create the new User objects, you could replace:

         user = new User(id, pair, msdSimilarUsers);

with

         user = new User(id, pair, new ArrayList());

If I've missed something (and you do have data in it before creating a User), create a new msdSimilarUsers ArrayList inside the while loop:

      while((line = inFile.readLine()) != null) {

        msdSimilarUsers = new ArrayList();

        tokenizer = new StringTokenizer(line);     // tokenize the file line
        // now add the user id to the user id array
 
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 2

Author Comment

by:override_y2k
ID: 9893735
Yeah what should happen is that the user should only be added to a users arraylist of similar users bif the mean squared difference between the users is less than 0.15. So msdSimilarUsers is only created after the whole file is read in, as only then are user profiles setup.We then need to calculate who the most similar users are and then add them into a users profile
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9893743
Oh sorry read that last comment wrong :)
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9893776
You're a star jimmack.Thx a million again for the solution.You've been a great help.Much appreciated!
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9893800
;-)
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9893809
Is there any particular reason why you're not asking questions in the Java area?
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9898267
I didnt even realise that i wasn't posting in the java area!

I actually encountered another slight problem with my system.

basically when i call the generateMsdprediction method it returns me the last value calculated for prediction(as the method processes every user).

However what i want it to do is to return the value for prediction that is calculated on each iteration of the loop.Obviously getting the same value for prediction every time completely changes the functionality.

Is there anywy in which i could do this?
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9900560
If I've understood you correctly, you actually want the results for each user returned.  If this is the case, you could do:

//    public static float generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+    public static ArrayList generateMSDPredictions(User user, int movieId, ArrayList userArrayList) {
+     ArrayList results = new ArrayList();
     float prediction = 0;
     float score = 0;
     float diff = 0;
     final float threshold = (float) 2.0;
     float weight = 0;
     Iterator iter = userArrayList.iterator();
     while (iter.hasNext()) {
       user = (User) iter.next();
       Iterator iter2 = user.getMsdSimilarUsers().iterator();
        while (iter2.hasNext()) {
          User userTmp = (User) iter2.next();
          diff = meanSquaredDifference(user, userTmp);
          weight = ((threshold - diff)/threshold);

          Iterator iter3 = userTmp.getpairArray().iterator();
          while (iter3.hasNext()) {
            Pair pair = (Pair) iter3.next();
            score = pair.getRatings();
            prediction = weight*score;
          }
         }
+         results.add(new Float(prediction));
       }
//       return prediction;
   }

(This lines with a + at the front are additions)
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9901086
The exact fucntionlity i want is that when i call this method in the addtraining method:

float predictedRating = generateMSDPredictions(userToTest, pairToTest.getId(), userArrayList);

should return 1 prediction for 1 movie.

So it should start by predicting what user 119 gave to movie 1.Retun this value to the addTraing method.
Then I subtract this prediction from the actaul rating, to give an error value.

Then predict what user 119 gave to movie 2, return this value,

Then what user 120 gave to movie 1 etc.

(Note: *user 119 and 120 are in user 1s arraylist of simUsers*)

Add up all the error values and divide by the number of predictions to get a mean error value.

I hope thats clear.If not let me know

0
 
LVL 15

Expert Comment

by:jimmack
ID: 9903469
I'm a bit pressed for time at the moment.  You should probably ask this as another question (since you've accepted my answer to your original question, no-one else is likely to comment in this thread now).
0
 
LVL 2

Author Comment

by:override_y2k
ID: 9906159
Oh ok thx.Will do so in the morning.Have been working on a solution, but am just muddling up the whole system now.
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Suggested Solutions

This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
Displaying an arrayList in a listView using the default adapter is rarely the best solution. To get full control of your display data, and to be able to refresh it after editing, requires the use of a custom adapter.
An introduction to basic programming syntax in Java by creating a simple program. Viewers can follow the tutorial as they create their first class in Java. Definitions and explanations about each element are given to help prepare viewers for future …
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now