?
Solved

endian conversion using algorithms and template

Posted on 2005-03-04
21
Medium Priority
?
426 Views
Last Modified: 2012-08-14

In order to convert a value from big endian to little endian, I could convert value to a native value, assemble the number, using the
proper multipliers.   So now - doing this from memory I could do:

 unsigned char big[4]; // single value in big-endian order  
  unsigned long x;
  x = ((big[0] * 0x100UL + big[1]) * 0x100 + big[2]) * 0x100 + big[3];

I'm sure there's a combination of algorithms and templates that could achieve this but that's where I'm lacking / hence asking for help converting from big to little and vice versa.  Something akin to:

template<typename T>
    T swap_endian( T begin, T end )
{
 // some algorithm .. maybe
}

int main()
{
}

0
Comment
Question by:forums_mp
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 10
  • 8
  • 3
21 Comments
 
LVL 6

Assisted Solution

by:guitaristx
guitaristx earned 225 total points
ID: 13460631
template<typename T> T swap_endian(T input)
{
     //assuming this is primitives only
     int s = sizeof(T);
     T out = 0;
     unsigned char* pos = &out;
     for(int i = s - 1; i >= 0; i--)
     {
          *pos = (unsigned char)(((unsigned char*)&input)[i]);
          pos++;
     }
     return out;
}

Basically, you're just treating input & out as though they are byte arrays, and swapping the bytes.  This will only work for primitive types, though.
0
 
LVL 30

Expert Comment

by:Axter
ID: 13460642
Hi forums_mp,
Are you trying to do this so that you can send data via socket?
If so, you can try using 'ntohs()', 'ntohl()', 'htons()', and 'htonl() functions.
See following link:
http://www.codeguru.com/forum/showthread.php?s=&threadid=298741

Cheers!
0
 
LVL 30

Expert Comment

by:Axter
ID: 13460657
forums_mp,
Also check out the following link for a wrapper function:

http://www.codeguru.com/forum/showthread.php?s=&threadid=292902
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:forums_mp
ID: 13461008
guitaristx:   I might have missed STL in my initial post but I'd like to use an STL algorithm.  If the answer is not plausible.   Then I understand.

Axter:
No, i'm on  a vxWorks platform.  I've got a powepc that uses big endian and a pci bus that's little endian.
so I send a

struct stuff {
 //
};

as data it arrives endian 'swapped'
0
 
LVL 30

Expert Comment

by:Axter
ID: 13461022
Did you look at the code in the last link I posted?

http://www.codeguru.com/forum/showthread.php?s=&threadid=292902
0
 

Author Comment

by:forums_mp
ID: 13461122

I did and my question becomes.  What if I'm not using the 'common' unsigned data type.   I'm sending structs.   Will this work?  
For some reason, I'm thinking that one solution would be std::bitset.  Admittidely,  I need to read more on bitset but it appears that I could treat the structs as a bucket of bits and convert them.
0
 
LVL 6

Expert Comment

by:guitaristx
ID: 13461371
Not easily, since C++ doesn't have an "easy" way to treat structs as aggregate types.  Are you sure that you understand the problem completely?  Let me give you a for-instance:

struct str
{
     int x;
     char y;
     char n;
     short z;
}

sizeof(str) should equal 8.  Now, let's see how that would look in memory:
(each byte of each member is separated by one space)
x1 x2 x3 x4 y n z1 z2 //on a big-endian system

This struct on a little-endian system would look like this:
x4 x3 x2 x1 y n z2 z1

where it seems that you're expecting it to be:
z2 z1 n y x4 x3 x2 x1

You can't use a template to automagically figure out what the members are, what their types and lengths are, and convert.  The run-time environment doesn't see a struct as anything more than a block of memory, but just reversing the bytes in the entire struct is going to yield gibberish.  You must convert the structs by-hand, unfortunately.  Templates can't help you here.
0
 

Author Comment

by:forums_mp
ID: 13461534
Sounds good.  I'll have to pull out the PCI bus analyzer again and review the data once more.  In the  interim,  thanks to you an axter.  The current support for floats doubles, will come in handy at some point.

Let me ask this.  Outside of templates and all that.   Wouldn't bitset be appropriate in terms of treating a struct as a bucket of bits, instead of doing it by 'hand'?
0
 
LVL 6

Expert Comment

by:guitaristx
ID: 13462029
It depends on why you need bitwise access.  Could you expound a bit more?  The main problem I see is converting your structs back & forth from bitsets.
0
 

Author Comment

by:forums_mp
ID: 13468211

guitaristx:
I'll have to get back with you on that - most likely monday.  I know there's an endian issue but I'll look at the bit pattern more closely.

///
On an entirely different issue.  Consider:

class X2{
public:
  X2() {}
  void compute(size_t len = 4096)
  {
    // check to ensure len is aligned on a 4k boundary
  }
};

// later
int main()
{
  X2 x;
  x.compute(10 );  // bad user passed 10..
                           // valid values 4096, 8092 etc. up to 16Meg max
}

Within the member function compute.  I'd like to validate that the passed paramater is aligned on a 4k boundary.   How would I achieve this?  I dont think my current approach is 'the best'.
0
 

Author Comment

by:forums_mp
ID: 13811953

Somehow I missed the fact that I had this open question.   guitaristx  here's my problem.  Which is duplicated in a different post but I think I'll close that other post now.

#ifndef  HEADER_H
#define HEADER_H

  struct msg_header_t {
     unsigned int  msg_id;
     unsigned int  msg_count;
     unsigned int  msg_size;
     unsigned int  msg_chksum;
  };
 struct t01_msg
 {
  signed int       xdircosine  :16;  // X Direction Cosine  - LSB 2^-15
  signed int       ydircosine  :16;  // Y Direction Cosine  - LSB 2^-15
  signed int       zdircosine  :16;  // Z Direction Cosine  - LSB 2^-15
    // more
 };

 struct outgoing_msg {
    msg_header_t msg_header_;
    t01_msg  t01_msg_;
    // more
 };

# endif

// header main_win.h

# include "header.h"
class main_win_impl : public main_win
{
  Q_OBJECT
  main_win_impl(QWidget *parent);
  ~ main_win_impl();

// later
private:
  QSocket         *socket;
  outgoing_msg  mPacket;
};

// main_win_impl.cpp
// later
#include <algorithm> //required for std::swap

#define ByteSwap5(x) ByteSwap((unsigned char *) &x,sizeof(x))

static void ByteSwap(unsigned char * b, int n)
{
   register int i = 0;
   register int j = n-1;
   while (i<j)
   {
      std::swap(b[i], b[j]);
      i++, j--;
   }
}

main_win_impl::main_win_impl(QWidget *parent)
  : main_win (parent)
{
  // later
  socket = new QSocket();
}

void main_win::send_message()
{
  if ( socket->state() == QSocket::Connected )
  {
     mPacket.msg_header_.msg_count  = mCount,  mCount++;
//   for (int idx(0); idx < sizeof(mPacket); ++idx)
//      ByteSwap5(  );
    socket->WriteBlock( (char *)&mPacket, sizeof(mPacket));
  }
}
0
 
LVL 6

Expert Comment

by:guitaristx
ID: 13825640
I think I see your problem.  The issue here is that you can't just byte-swap the entire struct, or you'll get gibberish.  You'll have to do it by-hand for each member of the struct, using the functions that Axter recommended:
>> You can try using 'ntohs()', 'ntohl()', 'htons()', and 'htonl() functions.
>> http://www.codeguru.com/forum/showthread.php?s=&threadid=298741

The fact that you're using bit fields doesn't really matter, since they're 16 bits long anyway.  I would recommend using signed short, rather than signed int, in your t01_msg struct:
struct t01_msg
 {
  signed short       xdircosine;
  signed short       ydircosine;
  signed short       zdircosine;
    // more
 };

Signed shorts are 16 bits in length, and it will simplify the conversions.  As far as swapping the byte-order, it should look like this for the above struct:

void toBigEndian(t01_msg* obj)
{
     obj->xdircosine = htons(obj->xdircosine);
     obj->ydircosine = htons(obj->ydircosine);
     obj->zdircosine = htons(obj->zdircosine);
     //more
}

I suppose that the million-dollar question is:
are you using bit fields that are not 8, 16, or 32 bits in length?  If not, you should be all set.  If so, then we've still got a bit more work to do.
0
 

Author Comment

by:forums_mp
ID: 13837501
>>>> I suppose that the million-dollar question is:
>>>> are you using bit fields that are not 8, 16, or 32 bits in length?

That's the thing.  I'm 'stuck' using definitions like this:

typedef double tag_word_t;

typedef enum
{
  sby                 = 1,
  a_to_g            = 2,
  a_to_a             = 4,
  bsigh               = 5,
  lns                   = 6
} sys_mode;

struct ts1_data
{
  tag_word_t       tag_wd;
  sys_mode         ofmode      : 4;
  bool                 ofiralg         : 1;
  // more
};

struct header_msg
{
  unsigned short size;
  unsigned short id;
  unsigned short count;
  unsigned short chksum;
};

struct outgoing_data
{
  header_msg hdr;
  ts1_data   t1_data_;
};

I'm trying to work with a host of solutions to this problem.   Haven't had a chance to really investigate but one approach I'd like to FIRST investigate is akin to:

For incoming/received messages.  I'll:
  1. allocate an unsigned char buffer large enough to hold the message.
  2. Read the message into the buffer
  3. Convert the ‘endianism’ of the data.
  4. Load each member of the model structure FROM the buffer.

For outgoing/transmitted messages.  I'll:
  1. allocate an unsigned char buffer large enough to hold the message.
  2. Load each member of the model structure INTO the buffer.
  3. Convert the ‘endianism’ of the data.
  4. Transmit message from the buffer


0
 
LVL 6

Expert Comment

by:guitaristx
ID: 13837987
Do you have control of both ends of the communication, or just one?  If you've got control of both ends, then you have the freedom to define the communication protocol however you see fit.  However, if you've only got control of one end of the communication, chances are that there exists a specification for the protocol somewhere.

Can you be a bit more specific about your problem?
0
 

Author Comment

by:forums_mp
ID: 13840142
I have control. The basic premise centers around the fact that I'm a client sending a server data.  because of 'endian issues' with client and server, one needs to handle endianness.  In that regard, I was given a plethora of advice with regard to what works best - more specifically 'portable code' -  in a situation like this.  The approach I thought about first investigating involves what's described above.

The claim is that 'char arrays' is portable/safe in this situation.  The one thing that confuses me about the approach descibed is.  If platform a (transmitter) is 16 bit integers.  platform b (receiver) is 32 bits
If I allocate a char array buffer large enough on platform a.  transmit the data to b.   b (prior to converting to the appropriate struct) sees a buffer thats 16 bits.  But b is 32 bit integers hence the buffer is not large enough.

Am I'm way off here?
0
 
LVL 6

Assisted Solution

by:guitaristx
guitaristx earned 225 total points
ID: 13842853
First, I think you've maybe believed that you can just dump your struct into a byte buffer and send it across the wire, which is a Bad Thing.  You're not guaranteed that the two platforms pack structs the same way, and this approach will cause you more headache than it will solve.

The best approach is to let the protocol between the two platforms determine the endian-ness and the format of the data.  What you'll need to do is determine which platform, a or b, is easier to do the byte-swapping (or just pick one).  Then, you'll tailor the protocol so that that platform (say, a, for example) _always_ transmits and receives data according to the endian-ness of the other platform.

[DISCLAIMER - the following is an explanation of how I would approach this problem.  It is not necessarily the "right way," just a way that could work]
Begin by examining the data structures on both sides.  (I believe they're declared the exact same way, right?)  Your protocol should be a "least common denominator" (for lack of a better term) of these data structures.
Create a data structure that will be the 'template' for how data is passed from one platform to the other.
Pad all values out to the nearest byte, 2 bytes, or 4 bytes -
1 bit becomes one byte, 4 bits becomes one byte, 13 bits becomes 2 bytes, 21 bits becomes 4 bytes.
Verify that both hosts read & write to this data structure using the same endian-ness (I'd use big-endian [motorola], since it's considered "network byte order").

Now that the data is defined, you don't really need the data structure we defined above (per se), since you've already got your native data structures.  Reading data from one host simply consists of reading the appropriate number of bytes, dealing with the endian-ness (as necessary), and placing it in the data structure that already exists.

For example:
Let's say that the following data structure (from above) is defined on both hosts:

struct ts1_data
{
  tag_word_t       tag_wd;
  sys_mode         ofmode      : 4;
  bool                 ofiralg         : 1;
};

The fields ofmode and ofiralg will both be padded to 1 byte each when they're transmitted to the other host.
So, whenever reading from one host to another, you'll read 8 bytes first, for tag_wd, and convert to the appropriate endian-ness (if necessary).  Then, you'll read one byte and place the value into ofmode (since no byte-swapping needs to occur for a 1-byte value).  Finally, read the last byte and place the value into ofiralg.
0
 

Author Comment

by:forums_mp
ID: 13854145

>>>>>>>>First, I think you've maybe believed that you can just dump your struct into a byte buffer and send it across the wire, which is a Bad Thing.  You're not guaranteed that the two platforms pack structs the same way, and this approach will cause you more headache than it will solve.

Come to think of it.  Perhaps I'm confused.  So again consider.

struct ts1_data
{
  tag_word_t       tag_wd;
  sys_mode         ofmode      : 4;
  bool                 ofiralg         : 1;
};

I'm not 'dumping' a struct.  I'm just dumping the contents of the struct.   IOW to dump the ts1_data struct to the buffer.   I envision.

   8 byte for tag_wd, dump 4 bits (ofmode), dump 1 bit (ofiralg).

Where I'm confused with this approach is:   ofmode and ofiralg can be combined in 1 eight bit 'buffer'.   Does each get their own 8 bit buffer or are they combined?

I think the answer though lies in a response I received when I first investigated this:  

If you always convert a double to 8 bytes in a known encoding, and the bools into a
certain sequence of bytes, then it should work...you're essentially
not just "copying the struct to the bytestream" but enforcing your own
encoding on it.

Now how does one 'enforce' their own encoding on 8bytes etc.

0
 
LVL 6

Accepted Solution

by:
guitaristx earned 225 total points
ID: 13858145
>> If you always convert a double to 8 bytes in a known encoding, and the bools into a
>> certain sequence of bytes, then it should work...you're essentially
>> not just "copying the struct to the bytestream" but enforcing your own
>> encoding on it.

This is a very good explanation.

>> Now how does one 'enforce' their own encoding on 8bytes etc.

I'll explain by example.  These are all for-instance examples, and are definitely not the ONLY way to do it:
"All C/C++ 'double' values will be transmitted in big-endian format using IEEE-754"
"All boolean values will be one byte in length, with zero indicating false and all other values indicating true"

What's important to remember is that you have complete control over the data transmission, but you must _exert_ that control.  If you can't explain what each bit of data represents, you're not enforcing an encoding.
0
 

Author Comment

by:forums_mp
ID: 13861019

After much investigation, I perused the web and found an implementation of ostringstream that's interesting.  So now consider:

#include <ostream>
namespace jngcomp {  namespace utils
{
     // Output string buffer class
     class stringbuf : public std::streambuf
     {
      public:
         stringbuf(char* buffer, int buflen)
             : m_buffer(buffer), m_buflen(buflen)
         {
             setp(m_buffer, m_buffer + m_buflen);
         }
         char* getpptr()
         {
             return pptr();
         }
         void setpptr(char* p)
         {
             setp(p, m_buffer + m_buflen);
         }
     protected:
         virtual int_type overflow(int_type c)
         {
             return EOF;
         }
         char* m_buffer;
         int m_buflen;
     };

     // Output string stream class which is suppled with a buffer to write to.
     // Ie, stream equivalent of sprintf.
     class ostringstream : public std::ostream
     {
     public:
         ostringstream(char* buffer, int buflen)
             : m_buf(buffer, buflen), std::ostream(&m_buf)
         {
         }
         char* getpptr()
         {
             return m_buf.getpptr();
         }
         void setpptr(char* p)
         {
             m_buf.setpptr(p);
         }
     protected:
         stringbuf m_buf;
     };
}}

int main()
 {
   test t_;
   t_.idx = 15;
   t_.jdx = 25;

   char buffer[10];
   jngcomp::utils::ostringstream strm(buffer, sizeof(buffer));
   strm << t_.idx ;
   std::cout << buffer[0] << std::endl;
   std::cout << buffer[1] << std::endl;
 }

What's interesting to me is the output is
1
5

With this approach how does one get 15 to spread across 4 bytes withouth having to do
strm << 0 << 0 << t_.idx; ?



      
0
 

Author Comment

by:forums_mp
ID: 13861051

By the way here's the test struct.

struct test {
   int               idx   : 32;
   unsigned int jdx   : 32;
   bool            kdx   : 1;
};
0
 
LVL 6

Expert Comment

by:guitaristx
ID: 13867697
I saw your other question:
http://experts-exchange.com/Programming/Programming_Languages/Cplusplus/Q_21402129.html

Is this question pretty much done?
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

What is C++ STL?: STL stands for Standard Template Library and is a part of standard C++ libraries. It contains many useful data structures (containers) and algorithms, which can spare you a lot of the time. Today we will look at the STL Vector. …
Many modern programming languages support the concept of a property -- a class member that combines characteristics of both a data member and a method.  These are sometimes called "smart fields" because you can add logic that is applied automaticall…
The viewer will learn how to user default arguments when defining functions. This method of defining functions will be contrasted with the non-default-argument of defining functions.
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.
Suggested Courses
Course of the Month8 days, 16 hours left to enroll

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question