Read IDX file to display DBF file!

IDX files are Index files. You cannot and you do not need to read its structure.

All of the Fox structures are very well documented.

Assuming you want to get into the .idx file, you have a couple of places to look.

(All) VFP File Structures
http://msdn.microsoft.com/library/en-us/dv_foxhelp/html/fooriFileStructures.asp?frame=true

Index File Structure (.idx)
http://msdn.microsoft.com/library/en-us/dv_foxhelp/html/conindex_file_structure_lpidxrp.asp?frame=true

OR

Compact Index File Structure (.idx)
http://msdn.microsoft.com/library/en-us/dv_foxhelp/html/conCompact_Index_File_Structure_LPidxRP.asp?frame=true

You can read everything in FoxPro. You an use Carl's links to get the structure and then write a small script to open the files in a low level format.

commands to use:

FOPEN
FREAD or FGETS
FCLOSE

Let me explain.

>> Every DBF file has a IDX file (index file)?,

No, that's not true. Only the files that have been indexed have an associated .IDX file. It works like this. When you create a database in FoxPro (actually a table in relational terms), it creates a .DBF file. When you index it, it creates a .IDX file. (Additional note: Foxpro can create more types of indexes).

>> My project at university is to read the IDX file and display the content of the equivalent DBF files.

I have not actually understood this. Can you please elaborate.

It is true that everything can be read. But reading at binary level is something no one would like to do untill absolutely needed.

I wouldn't recommend directly opening .IDX files with FOPEN (untill that's the only way out).

FoxPro's default index is a .CDX file which is a compound index. Think of it as a multiple IDXs. Usually if something is wrong in the index in FoxPro we simply recreate it from the .DBF.

USE file EXCLUSIVE
DELETE TAG ALL
INDEX ON field1 TAG field1
INDEX ON field2 TAG field2
USE

This will recreate or create .CDX from a .DBF.

If you need to study the internal structure of an .IDX file, you will need to open it using FOPEN. Once I created a dbf recovery program and I needed the structure of a DBF and FPT file. If this is not what you desire, than you can open the index file like this:

USE file INDEX index.idx
SET ORDER TO fieldindexed
SCAN ALL
    ? fieldindexed
ENDSCAN
USE

Its easier to read the structure of a DBF file from the DBF itself. The structure of a DBF file is stored right at the beginning, before its data part begin. Back in the days of dBase III+ and FoxBase+ I had written a program in C to recover data from a corrupted DBF file using the same method.

yes I agree, RanheetRain, and that's what I did. But I am assuming tinh911 wants to read how the index keys are stored in the IDX file and how to find the record number from there.

You just get the file structure from Carl's links. I did not see them personally but I am sure they are the ones, or maybe from the help in FoxPro.

Open the IDX in low-level format and get the keys as well as the B-Tree structure and the record numbers. This is as far as I can help you.

This is from the help in VFP 6.0

Compact Index File Structure (.idx)
See Also

Compact Index Header Record

Byte offset Description
00 – 03 Pointer to root node
04 – 07 Pointer to free node list ( -1 if not present)
08 – 11 Reserved for internal use
12 – 13 Length of key
14 Index options (any of the following numeric values or their sums):
1–a unique index
8–index has FOR clause
32 –compact index format
64 –compound index header
15 Index signature
16 – 19 Reserved for internal use
20 – 23 Reserved for internal use
24 – 27 Reserved for internal use
28 – 31 Reserved for internal use
32 – 35 Reserved for internal use
36 – 501 Reserved for internal use
502 – 503  Ascending or descending:
0 = ascending
1 = descending
504 – 505 Reserved for internal use
506 – 507 FOR expression pool length1
508 – 509 Reserved for internal use
510 – 511 Key expression pool length1
512 – 1023 Key expression pool (uncompiled)


1 This information tracks the space used in the key expression pool.

Compact Index Interior Node Record

Byte offset Description
00 – 01  Node attributes (any of the following numeric values or their sums):
a.0 – index node
b.1 – root node
c.2 – leaf node
02 – 03 Number of keys present (0, 1 or many)
04 – 07 Pointer to node directly to left of current node (on same level, -1 if not present)
08 – 11 Pointer to node directly to right of current node (on same level; -1 if not present)
12 – 511 Up to 500 characters containing the key value for the length of the key with a four-byte hexadecimal number (stored in normal left-to-right format):
This node always contains the index key, record
number and intra-index pointer.2

The key/four-byte hexadecimal number combinations will occur the number of times indicated in bytes 02 – 03.
 


Compact Index Exterior Node Record

00 – 01  Node attributes (any of the following numeric values or their sums):
0 – index node
1 – root node
2 – leaf node
02 – 03 Number of keys present (0, 1 or many)
04 – 07  Pointer to the node directly to the left of current node (on same level; -1 if not present)
08 – 11  Pointer to the node directly to right of the current node (on same level; -1 if not present)
12 – 13 Available free space in node
14 – 17 Record number mask
18 Duplicate byte count mask
19 Trailing byte count mask
20 Number of bits used for record number
21 Number of bits used for duplicate count
22 Number of bits used for trail count
23  Number of bytes holding record number, duplicate count and trailing count
24 – 511 Index keys and information2


2 Each entry consists of the record number, duplicate byte count and trailing byte count, all compacted. The key text is placed at the logical end of the node, working backwards, allowing for previous key entries.

tinh911,
Maybe you don't need the structure itself at all. Check foxpro api. It has a lot of ready functions that you can use from C (ie: _dbseek,_dbwrite,...).

BTW to read a dbf you don't need its index file(s).

All of those functions to read .dbfs and .idx files, etc., are already written and done in C in the CodeBase C library that many C developers use to get to Fox data.  I worked with a C programmer about 12 years ago who used exactly that library back when we were still in FoxPro for DOS and he was working in the Microstation CAD environment using data interactively from and to FoxPro for DOS.  The CodeBase library worked great for him.

CodeBase (by Sequiter Software, Inc.)
http://www.sequiter.com/products/

FoxPro, dBASE & Clipper File Compatibility: A High-Performing Standard
****************************************************
CodeBase-powered applications are multi-user file compatible with FoxPro, dBASE and Clipper (xBASE). You can read and write the data, index and memo files of xBASE at the same time as xBASE.

Enjoy great performance using xBASE files for several reasons. First, the internal xBASE algorithms are simple and efficient, and the database files compress well which minimizes disk I/O. The format also lets you split a database across different disk drives allowing you to increase disk I/O throughput as much as necessary. Finally, operating system disk defragmentation utilities work excellently on xBASE files which allows maximum efficiency.

The xBASE database format is one of the most popular in the world. Thousands of software products either use or import/export data in the xBASE format. Consequently, you can use CodeBase with many development tools and applications.

SQL programmers will appreciate the unique record number field with super fast record retrieval (no index is required) and no storage overhead. One last benefit for xBASE programmers is that the corresponding API option will be intuitive and easy to use.

There is a freely downloadable CodeBase speed demo at the following URL:

http://www.sequiter.com/downloads/

Well, I wouldn't say the data is not important.  

But, as you say, you are concentrating on the index scheme of things because that is your assignment.

If you are looking at keys that are not just GUID references or sequential primary keys with no meaning, then the data part is of little use in this context.  However, if for other reasons you are indexing (or someone else has indexed) on other columns in the data that are a bit unusual and you get unusual results in your idx functions, you may have to look at the source data to see why the idx file is the way it is.  

So, I guess, as usual, it depends.

I don't see the point why you need to do that. DBF file is important there (each key value also has a pointer to record) and Fox does that when you add,delete, modify records.
If you strip doing IDX updates from DBF itself you'd be travelling back in history when IDX files were separate from their DBF and users were easily falling into mistake not to open and update them too.
If you'd still do Foxpro indexes are somewhat B Tree indexes. Check structure.

<< Where can I get the source of C Library of CodeBase to do my jobs! >>

Other than purchasing it, I really don't even know who has it currently.  It's been a long time since I worked with that C programmer who had access to it and I don't even know where he works today.

Maybe some searching on the 'net will get you in contact with someone who is actively using it.

IMHO codebase is old for the job. Structures has gone well modification since that (OTOH I don't know if has newer releases that can handle those changes)

If you have VFP read IDX file structure (and compact index file structure) sections on online help or from MSDN help. For a C coder they're quite easily understandable.

You might do a simple trick and read only the key expressions, build the tree yourself, rather than trying to sort out from IDX. With multiple tags in a single file it might get harder to sort out from IDX itself.

PS: Recently few use IDX files but CDX.

Dear tinh911,

you can do the code yourself. Just use the structure that I copied for you from VFP Help. I did it for the dbf and fpt. It's just the structure you have to decipher.

You can start by opening the idx in lowlevel format and then printing the keys and following the index to the next node. Once you got that done the rest is easy to add and delete keys.

I did that once in C back in 1990. It was my assignment as well. But unfortunately I don't have the code. I left the code in the States. I did it using double level and triple level pointers in C. That's what I remember from back then. It was fun.

As per the Codebase, try www.sequiter.com and www.codebase.com.
As I could see their page is new and seemingly supporting newer VFP format.
Mine version is old and on the shelf in dust at home.
I sorted out as Captain said and wrote my own routines mostly in foxpro and  some in C (compiled as a plb and fll then but never did it for the idx itself as I could simply delete and recreate all the tags).

Thanks and you're welcome.  Good luck with your project.