Link to home
Start Free TrialLog in
Avatar of HLRosenberger
HLRosenbergerFlag for United States of America

asked on

Overlay data type

I am trying to reverse engineer a DBF and how it is used an application.  The application has a number of date fields.  I can see these date fields in the certain columns of the DBF.  What's strange is the that dates exist across multiple rows in the DBF.   if I extract the data for all the columns and concatenate into one long string, I see the exact dates I'm expecting, as they appear in the application.   I'm using a product called DBF Viewer to look at the DBF file, as I don't own a full blown version of Visual Foxpro.  To extract the data and concatenate it, I'm using VB.NET code and FoxPro drivers.  

The column names that contain the dates do not indicate that they are dates. To the contrary, the names are totally unrelated.  So my question is this - can multiple data columns in a DBF be "overlayed" and used as one large text data column by the application?   I know certain lower level programming languages have this concept.
Avatar of Pavel Celba
Pavel Celba
Flag of Czechia image

No, DBF file does not allow columns overlaying. OTOH, the date data type is stored as a text value in DBF and you can also use character data type to store the date. To store date in a character column brings certain variability to programmers (e.g. incomplete date entry) but also requires more work to handle such character data.

DBF viewers should display the structure and appropriate data types for all columns.

You could even use Visual FoxPro Run-time to display and manipulate your DBF data.
Just download the installer: http://www.foxpert.com/runtime.htm
and VRunFox 9 application: http://leafe.com/dls/vfp
This approach seems to be legal but you have to know VFP instruction set. VB.NET code and the driver is also OK.

BTW, one VFP function - SYS(2017) - uses DBF row as one text line.
Avatar of HLRosenberger

ASKER

Thanks for the info.  To expand on my question. The app has 64 dates, 8 groups of 8.  They are immunization dates for children.  See images.  Some of the columns I see having this date data are defined as numeric fields, but most are text data.   None of the columns are defined as date, and I do not believe the app "thinks" of them as dates, but instead it's just text entry.   Bottom line is across 20 columns, if I pull the data and concatenate it, it matches to the dates I see in the app.  See images.    

So, this is doable then?

User generated imageUser generated imageUser generated image
Avatar of jrbbldr
jrbbldr

I am trying to reverse engineer a DBF and how it is used an application

First question - why?
* Are you doing this as a 'learning project'?
* Or are you trying to 'reverse engineer' an application so that you can do something with it?

If the former, great.
If the later, do you or your company legally OWN the code?  If not, there might likely be some legal questions involved about what you are trying to do.  
The Data itself might be yours, but how the application results in the data may not be.
If you do OWN the code then you should be studying the application's code for answers.

After having to interface new code packages into existing applications where I have only had access to the data and not its code, I can tell you that you most likely will be facing a rather large challenge.
It is one thing to get to a point of understanding what each field in a data table represents, but understanding how and when that field gets populated is often quite a different issue.
And if multiple data tables are involved the project gets quite a bit 'muddier'.

Your best bet is to try to understand how the code works when certain operations are performed - what data table fields are modified - when and how.  Then, from that you can better understand how the fields and their associated data relate to the application's utilities.

Good Luck
Neither, really.    We own the data.   We have a license for the application.   We don't own the source code.  I'm trying to "reverse engineer" as far as where these dates are stored in the DBF.    We are writing a .NET application, using FoxPro drivers, to access the data for our own internal use.    It seems bizarre that the dates are spread across these columns, but it does seem like just that.   So, some mechanism must have be used to do this.  I am not interested in the mechanism, or in try to replicate it, but merely in reading the dates from VB.NET.   And I want to be sure I have the correct data, since again this seems to  bizarre.
I cannot confirm "the data are stored as texts"... If this is true then why do you use ToString() function to concatenate them? But I also cannot confirm the data are stored as date columns or numbers...

The code used for concatenation works on 20 columns whereas the output shows just 8 dates...

I would recommend following code (please fix the syntax as I don't use VB.NET)
1. List the structure:
for n = 0 TO Reader.Items.Count - 1      ...   VB.NET maybe need  for n= 1 to Reader.Items.Count
  Console.WriteLine(Reader.Item(n).Name)
next
2. List data types
  Reader.Item(n).DataType  ... intellisense will tell the correct property name

Then you may assign values to column names, then you may decide what to do next.

Could you send the DBF file itself? Or could you post printscreen from DBF viewer?
I can't sent you the DBF.  HIPPA rules apply.   I changed the VB code to print out column names and data types.   I attaColumns.txtched
OK, so you may check the data values in all rows and try to use them now.

It seems some date values are stored as dates but I can imagine some vaccination can be stored as a number of days related to the previous date etc. And the output also points to many missing values in DBF. The software could have a reason for this variability or it just reflects development team experience and fluctuations and you are the one who has to decode it now... And remember next software version can use different column names...

The best you can do is to ask the software vendor for some API or export format availability.
how it is used an application
I'm trying to "reverse engineer" as far as where these dates are stored in the DBF.

OK, what you are attempting to do sounds like it is legal, but have you asked the application vendor if what you want to is OK'd by them?
Many times manipulating the data from another vendor's application will quite often violate any support agreement that you may have with them?  They don't want other people/applications messing with the integrity of the data that they write so as to prevent any possible data referential integrity issues from arising, thereby causing their original application to misbehave and/or crash.

On a side note...
Out of curiosity I just now opened a Copy of a VFP9 Test DBF file that I had into a Hex Editor.
That Test DBF had a variety of Field types in it.
The Hex Editor I used displayed both the Hex value and the ASCII representation of the bytes
I looked for a specific field name that I knew was there and I found, not only it appearing in the ASCII, but it was followed by the Field Type (4 bytes after the last char of the field name).
C --> Character
D --> Date
L --> Logical
etc.
You might want to give that a try (on a COPY of the data table(s)) to more effectively determine the field type(s).

Regardless, I'd suggest that rather than trying to find out solely from the data-side, you try to look at things from the application side.  That is how I have effectively (albeit slowly) approached things like this in the past.

1. Create a 'dummy' client whose data you will manipulate for testing only purposes.
2. Within the application, do a function.
3. Use the DBF file's modify dates to see what tables were changed by that action.
4. Now examine that client's data in all of the changed tables to see what was changed and where.
5. Then 'map' those data field(s) and its associated function.
In that way you will better understand how specific fields get populated and when - thereby providing you with a more comprehensive understanding of the data and its possible referential integrity issues with other data fields and/or data tables.

The single data string you have generated via your VB operation may contain all of the screen form data, but without knowing how/when it is put into the data table(s), you are left with merely a string of characters (some of which 'look' good) and no associated understanding - nothing more.  

Good Luck
I'm not manipulating the data.  Only reading it; that's all I want to do.  it seems odd that taking, into account the column names, the application is storing array of dates across those columns.   And there are missing dates in the data, and those correspond to missing dates on the application screen, which reinforces that the application is indeed storing the dates in those columns.
And why would I have to ask the vendor anything as far as permission, when I'm only reading the data.
To have guaranteed output format is always better than your own investigations which may be inaccurate and which may become obsolete with new software version.

If you decided to decode/hack the data format then it is your decision and I cannot judge whether it is legal or not as I don't know all conditions... OK, data are yours probably.

So, you can read the DBF data, you know DBF structure, and you know dates can be stored/recalculated many different ways as several different data types in the application.

And what is the question now? How can we help further? As we don't know your data except one 8x8 printscreen we cannot tell how the application calculates this matrix but it seems this matrix is stored in one DBF table row.

You should extend the structure attached as columns.txt in previous post by values from one DBF row. Something like:

Column name = account ; Column Type = DBTYPE_NUMERIC ; Column Value = 0
Column name = name ; Column Type = DBTYPE_CHAR ; Column Value = "SomeName"
Column name = birthday ; Column Type = DBTYPE_DBDATE ; Column Value = 01/29/2016
etc.

OK?
why would I have to ask the vendor anything as far as permission, when I'm only reading the data.

OK, it originally sounded as though you were going to have your VB application possibly Read & Write the data.   If you are only Reading, then OK, the vendor might not need to give their OK.
NOTE - part of our ability to best advise you is to understand as much about your project as we can.

However I still think that just because you can recognize a Date value (whether from a Date Field or a Character Field) is not enough understanding of what that Date represents.   Therefore I still feel that you need to improve your understanding by testing from the application side.

Either you think you already know enough or you don't.
Regardless, Pavel is correct.   What is your question?  Either you have access to the information you need or not.

Good Luck
I have to say, that application screenshot is pretty cryptic.  Doesn't this application explain what these data elements are and what they're used for, on other screens or in documentation or somewhere?  How do those values get into the table?
What GEOFS says above is what I have been saying all along.

Seeing a Date in a screen form and/or finding a recognizable Date in a string of data without understanding how, when, or why it got there is of little or no value.

But if you think you have an adequate level of understanding - fine.

Otherwise please clarify your question so that we can better assist you.

Good Luck
I understand all that.   Bottom line is I wanted to know is this situation was possible, and there was some valid "overlay" mechanism.
ASKER CERTIFIED SOLUTION
Avatar of jrbbldr
jrbbldr

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Bad choice of words on my part for my original question.

Thanks for your help.