Java buffer, objects, ListArrays and converting String to ListArray

I am having trouble with some code that is written using a combination of Talend and my own custom coding.  I don't believe you need to know anything about Talend in order to help me out with this question.

Talend has a concept called a "job" which is essentially a java program which performs some user defined operations.  One job can call another and can pass parameters into the child job.  There is only one way that a child job can pass information back to the calling job, and that is through the buffer.  I am having trouble with a complex type that I am trying to pass back to the calling job via the buffer.  The type I am referring to is an ArrayList of type "Dynamic" (Dynamic is a custom class).

Essentially here is what I have:

In the Child Job:
---------------------------------------------------------
List<Dynamic> genericData = new ArrayList();  
   .. then there is a loop here:
        genericData.add(row2.generic_data.clone());
   .. end loop

globalMap.put("genericData", (Object)genericData);

then write this to the buffer:  ((Object)globalMap.get("genericData"))

Talend does not give me the ability to write a type of List<Dynamic> to the buffer, so I need to cast it as an Object first.
--------------------------------------------------------
In the calling job:
--------------------------------------------------------
List<Dynamic> myData = new ArrayList();
myData = (ArrayList)input_row.dataObject;

When I run this, I get this error:  

java.lang.ClassCastException: java.lang.String cannot be cast to java.util.ArrayList

So it appears that the datatype of the object has been changed from List<Dynamic> to string, and I cannot figure out how to convert this back into a type of List<Dynamic>.

I think the program flow should be like this:

1. Inside the child job, create a variable of type of List<Dynamic>.  Fill it with data.
2. Convert the variable to type Object and write it to the buffer.
3. In the calling job, get the Object from the buffer, and convert it from Object to List<Dynamic>.  

Step 3 is where the problem is.  For some reason, the variable is type java.lang.String after reading from the buffer.  

How can I convert this back to List<Dynamic>?

Thanks.
jbaird123Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

CEHJCommented:
0
mrcoffee365Commented:
Usually collections can't just be "cast" to one another.  Normally you loop through one collection and add its elements to the new collection type.  So try looping through your List<Dynamic> and doing the put (add?) to the ArrayList object you're trying to fill.
0
jbaird123Author Commented:
CEHJ - I will post the stack trace momentarily.

mrcoffee365 - I don't think you understood my issue.  I can't loop through my List<Dynamic> because that object gets converted to a string when I pass it through the buffer.
0
Exploring SQL Server 2016: Fundamentals

Learn the fundamentals of Microsoft SQL Server, a relational database management system that stores and retrieves data when requested by other software applications.

jbaird123Author Commented:
CEHJ:

Here is the full console output from my job:

Starting job GET_SOURCE_DATA_JB_test at 14:12 21/10/2013.


[statistics] connecting to socket on port 3401
[statistics] connected
2013-10-21 14:12:04 : INFO : Processing database / table myConnection / WMS_IN_BUSINESS_PARTNER...
Exception in component tJavaRow_7
java.lang.ClassCastException: java.lang.String cannot be cast to java.util.ArrayList
      at testjob.get_source_data_jb_test_0_1.GET_SOURCE_DATA_JB_test.tRunJob_6Process(GET_SOURCE_DATA_JB_test.java:1384)
      at testjob.get_source_data_jb_test_0_1.GET_SOURCE_DATA_JB_test.tMSSqlConnection_1Process(GET_SOURCE_DATA_JB_test.java:1111)
      at testjob.get_source_data_jb_test_0_1.GET_SOURCE_DATA_JB_test.runJobInTOS(GET_SOURCE_DATA_JB_test.java:2009)
      at testjob.get_source_data_jb_test_0_1.GET_SOURCE_DATA_JB_test.main(GET_SOURCE_DATA_JB_test.java:1482)
2013-10-21 14:12:05 : INFO : Fetched 11 rows into memory from null...
row3.dataObject.getClass: class java.lang.String
[statistics] disconnected
Job GET_SOURCE_DATA_JB_test ended at 14:12 21/10/2013. [exit code=1]
0
CEHJCommented:
Could you also please attach the source of GET_SOURCE_DATA_JB_test.java
0
jbaird123Author Commented:
CEHJ: The source is attached.  99% of this code is created by Talend, but there are a couple of places where I do custom coding.

Line 674 is where it fails:  

                              myData = (List<Dynamic>) row3.dataObject;
GET-SOURCE-DATA-JB-test.java
0
CEHJCommented:
  at testjob.get_source_data_jb_test_0_1.GET_SOURCE_DATA_JB_test.tRunJob_6Process(GET_SOURCE_DATA_JB_test.java:1384)

Open in new window

is where the problem occurs. But that wasn't the correct source, as there are only slighly over 1000 lines in the source. So there is no line 1384
0
jbaird123Author Commented:
CEHJ:  I'm not sure why that last file didn't work.  I created a simpler job and attached the code.

This job just creates some data and loads it into the buffer and then reads it from the buffer.

If you look at line 959 you will see this:

                        List<Dynamic> myData1 = new ArrayList();
                        myData1 = (List<Dynamic>) globalMap.get("genericData");

This code works and is ALMOST the same as the code that fails later on.  The code that fails is at line 1403:

                              List<Dynamic> myData = new ArrayList();
                              myData = (List<Dynamic>) row1.dataObject;

The only difference is that the first code is assigning a value to myData1 BEFORE the data has been placed in the buffer.  The "bad" code is assigning a value to myData AFTER reading the buffer.
DB-GET-DATA-2.java
0
CEHJCommented:
I need to see the pair please:

a. stack trace
b. source file
0
jbaird123Author Commented:
Sorry.  Below is the stack trace.  Is the file I just gave you enough?

C:\temp\DB_GET_DATA_2_0.1\DB_GET_DATA_2_0.1\DB_GET_DATA_2>java -Xms256M -Xmx1024
M -cp classpath.jar; testjob.db_get_data_2_0_1.DB_GET_DATA_2 --context=Default

row1.dataObject.getClass: class java.lang.String
Exception in component tJavaRow_1
java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List
        at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tBufferInput_1Process(DB_GET_
DATA_2.java:1461)
        at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tFixedFlowInput_3Process(DB_G
ET_DATA_2.java:1267)
        at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tFixedFlowInput_2Process(DB_G
ET_DATA_2.java:1035)
        at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tSetGlobalVar_1Process(DB_GET
_DATA_2.java:470)
        at testjob.db_get_data_2_0_1.DB_GET_DATA_2.runJobInTOS(DB_GET_DATA_2.jav
a:1696)
        at testjob.db_get_data_2_0_1.DB_GET_DATA_2.main(DB_GET_DATA_2.java:1557)


C:\temp\DB_GET_DATA_2_0.1\DB_GET_DATA_2_0.1\DB_GET_DATA_2>
0
mccarlIT Business Systems Analyst / Software DeveloperCommented:
In your original post above you write...
then write this to the buffer:  ((Object)globalMap.get("genericData"))
I think this is where the problem is. You are passing the object to the buffer, but when you get the return value from the job it is a String[][]. So what code implements that buffer, is doing some conversion from Object -> String (or Sting[] or String[][]). Either way, I am guessing that that conversion is just simply using Object.toString() and so the return value just wont have the information that you need to reconstruct your List<Dynamic> object.

Can you test my suspicions by doing a System.out.println(row3.dataObject); just before you attempt the cast to List<Dynamic> and then post back here what it prints out?

If I am thinking correct, and there definitely is NO other way to pass information back, you probably will need to look at serializing your List<Dynamic> to something that you can deserialize from, such as using a JSON representation.
0
jbaird123Author Commented:
CEHJ:

Sorry for the delay in responding.  Below is the output you requested.  It looks like the data is still there, I just can't seem to convert it back.

If there is no way to convert it back to Dynamic, is there a simple command I can use to write the object to the buffer myself to avoid any conversion to String in the first place?

Thanks.

Starting job DB_GET_DATA_2 at 11:00 22/10/2013.


[statistics] connecting to socket on port 3415
[statistics] connected
Exception in component tJavaRow_1
java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List
      at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tBufferInput_1Process(DB_GET_DATA_2.java:1478)
      at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tFixedFlowInput_3Process(DB_GET_DATA_2.java:1273)
      at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tFixedFlowInput_2Process(DB_GET_DATA_2.java:1021)
      at testjob.db_get_data_2_0_1.DB_GET_DATA_2.tSetGlobalVar_1Process(DB_GET_DATA_2.java:416)
      at testjob.db_get_data_2_0_1.DB_GET_DATA_2.runJobInTOS(DB_GET_DATA_2.java:1711)
      at testjob.db_get_data_2_0_1.DB_GET_DATA_2.main(DB_GET_DATA_2.java:1578)
dataObject: ["Aaaaa" - "Aaaaa" - "Aaaaa", "Bbbbb" - "Bbbbb" - "Bbbbb", "Ccccc" - "Ccccc" - "Ccccc"]
[statistics] disconnected
Job DB_GET_DATA_2 ended at 11:00 22/10/2013. [exit code=1]
0
mccarlIT Business Systems Analyst / Software DeveloperCommented:
By the way, I'm not CEHJ  ;)

Can you post the source code for your Dynamic class?

is there a simple command I can use to write the object to the buffer myself to avoid any conversion to String in the first place?
No, the return type from running the job is a String (well a String array, but you get the idea) so there is NO way of getting anything other than a String out.

Though it looks like it might be easy to reconstruct your Dynamic list from the string representation given, so that will probably be the easiest road to a solution. But yeah, I would need the code for Dynamic to help you do this.
0
jbaird123Author Commented:
mccarl - sorry for the confusion!  :)
Dynamic.java
0
jbaird123Author Commented:
A couple of other files are also attached in case you need them.

Thanks for your help with this!
DynamicMetadata.java
DynamicUtils.java
0
mccarlIT Business Systems Analyst / Software DeveloperCommented:
Ok, so Dynamic is quite a bit more complicated than is apparent from the String that is in dataObject, and it would be impossible to fully construct the Dynamic object again from that string representation. So we have to do things another way. Fortunately, Dynamic and DynamicMetadata both implement Serializable so we can use this fact to build a better String representation of your List<Dynamic> object inside the child job, return that String basck and then when you get the return value String, we can deserialize it back to a List<Dynamic> object.

The following code should be fairly self explanatory from the comments, method names, etc. You can use the two static methods in here, in your code, but it is also a self-contained runnable example if you just want to check what it is doing...
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;

import javax.xml.bind.DatatypeConverter;


public class TestSerialization {

    public static void main(String[] args) throws IOException, ClassNotFoundException {
        List<Dynamic> list = new ArrayList<Dynamic>();
        
        list.add(new Dynamic("aaa"));
        list.add(new Dynamic("bbb"));
        list.add(new Dynamic("ccc"));
        
        
        
        // Inside the child job...
        
        // This is what we start with
        System.out.println("BEFORE:  " + list);
        System.out.println();
        
        // Serialize it
        String serializedString = serializeDynamicListToString(list);
        // Now pass serialized string to the "buffer"
        
        
        
        
        
        // Just to show what it is doing
        System.out.println("Serialized version:  " + serializedString);
        System.out.println();
        
        
        
        
        
        
        // Now outside the child job...
        
        // Get the string from the return values (in this test case we are just using the serializedString from above
        String row3_dataObject = serializedString;
        
        // Deserialize it
        List<Dynamic> newList = deserializeStringToDynamicList(row3_dataObject);
        
        
        // Just to show that it worked
        System.out.println("AFTER:  " + newList);
    }
    
    
    
    
    
    
    
    // Put these two methods somewhere (DynamicUtils.java for instance) so they can be used both inside and outside the child job
    
    public static String serializeDynamicListToString(List<Dynamic> dynamics) throws IOException {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ObjectOutputStream oos = new ObjectOutputStream(baos);
        
        oos.writeObject(dynamics);
        oos.close();
        
        return DatatypeConverter.printBase64Binary(baos.toByteArray());
    }
    
    @SuppressWarnings("unchecked")
    public static List<Dynamic> deserializeStringToDynamicList(String serializedString) throws IOException, ClassNotFoundException {
        ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(DatatypeConverter.parseBase64Binary(serializedString)));
        return (List<Dynamic>) ois.readObject();
    }
    
    
    
    
    
    
    
    // This is just a "test" Dynamic class so that this example runs properly, you obviously don't need this in your code!
    private static class Dynamic implements Serializable {
        
        private static final long serialVersionUID = -8796848952339474003L;
        
        private String testString;

        public Dynamic(String testString) {
            this.testString = testString;
        }

        @Override
        public String toString() {
            return "Dynamic [" + testString + "]";
        }
    }
}

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
jbaird123Author Commented:
mccarl - This solution is perfect!  Thank you very much for your patience on this.  I have been struggling with this for a few days now, so your help is very, very much appreciated.
0
mccarlIT Business Systems Analyst / Software DeveloperCommented:
You're welcome! ;)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java

From novice to tech pro — start learning today.