?
Solved

Data Acquisition Problem

Posted on 2004-10-21
19
Medium Priority
?
326 Views
Last Modified: 2011-09-20
I wrote (with a great help from drichards) a program in Visual C++ .NET to acquire data from a PIC device via the RS232 port (under Windows XP Professional). In general it seems to work fine except for a recurring annoying problem. The program is set to acquire 90 points in time. As the acquisition progresses sometimes the point acquired is faulty – usually one or two points out of 90. Interestingly, the errors are reproducible – the faulty points are the same upon subsequent running of the program. I should mention also that, if the hyperterminal is used to acquire the data from the PIC no such errors are observed. It seems that the additional calculations which my program performs to crunch the acquired data and display it graphically on the screen somehow interfere with the rhythm of the acquisition of these data and a moment comes when the data sent by the device cannot be processed by the VC++.NET program. Based on this supposition I tried to play with the timeouts in the VC++.NET program as well as with the various timings in the program of the PIC and could reach some success – if most processes running in the background are stopped (via the Task Manager) the faulty points disappear. However, if now I start again certain programs (such as, say, Yahoo messenger) and they run in the background the faulty points begin to appear reproducibly again and what exactly these faulty points will be depends on what particular program is running in the background.

Is there a way to make sure that the data which the VC++.NET program is supposed to process is always properly acquired by that VC++.NET program (despite what other programs are running in the background)?

The code I use now for data acquisition is the following:


#include "StdAfx.h"
#include "./multimeter.h"
#using <mscorlib.dll>

using namespace Heat;


MultiMeter::MultiMeter()
{
    m_mgdBuf = new unsigned char __gc[32];
}


void MultiMeter::Close()
{
    CloseHandle(m_hComm);
}


PASS_FAIL MultiMeter::ConfigureSerialInterface()
{      
      MultiMeter::Close();
      // 22-812 serial rs-232 interface code . . .
      //
      DCB config;
      //
      // open serial port
      //
    m_hComm = 0;
      System::Text::StringBuilder *port = new System::Text::StringBuilder(S"COM");
      System::Text::ASCIIEncoding *enc = new System::Text::ASCIIEncoding;
      unsigned char portChars __gc[] = enc->GetBytes(port->ToString());
           unsigned char __pin *portName = &portChars[0];
      
try
          {
      m_hComm = CreateFile(      "COM1",
                                    GENERIC_READ | GENERIC_WRITE,
                                    0,
                                    NULL,
                                    OPEN_EXISTING,
                                    0,
                                    NULL);
            
    portName = NULL;      
       }
      catch (Exception *exception)
         {
         }      
      if (m_hComm == INVALID_HANDLE_VALUE)
      {
            //log(plogfile, "Could not COM port (error)");
        return FAIL;
      }
      //
      // get port state and store in DCB
      //
      if (0 == GetCommState(m_hComm, &config))
      {
      //log(plogfile, "Error in getcommstate");
            return FAIL;
      }
      //
      // configure port
      //      
      config.BaudRate = CBR_2400;      
      config.Parity = NOPARITY;
      config.ByteSize = 7;      
      config.StopBits = TWOSTOPBITS;
      //
      if (0 == SetCommState(m_hComm, &config))
      {
            //log(plogfile, "Error in setcommstate");
            return FAIL;
      }
      //
      // set timeouts
      //
    COMMTIMEOUTS comTimeOut;                  
    comTimeOut.ReadIntervalTimeout = 0;
    comTimeOut.ReadTotalTimeoutMultiplier = 0;
   comTimeOut.ReadTotalTimeoutConstant = 2000      comTimeOut.WriteTotalTimeoutMultiplier = 0;
      comTimeOut.WriteTotalTimeoutConstant = 0;      
      //
    SetCommTimeouts(m_hComm,&comTimeOut);
      //
      return PASS;
}


System::String* MultiMeter::ReadString()
{      
    DWORD dwRead = 0;
    System::String *result;

    BYTE *pBuf = NULL;

    unsigned char __pin *buf = &m_mgdBuf[0];
    dwRead = ReadPort(buf, 27);
    buf = NULL; // We're done with pinned pointer

        if ( dwRead > 0 )
          {
        result = System::Text::Encoding::ASCII->GetString(m_mgdBuf, 0, dwRead);
          }
        else
          {
        result = S"";
          }
    return result;
}


int MultiMeter::ReadPort(unsigned char *buf, int buflen)
{
      FlushFileBuffers( m_hComm );
      DWORD dwRead = 0;
      if ( ReadFile( m_hComm, buf, buflen, &dwRead, NULL ) == FALSE ) dwRead = 0;
    return dwRead;
}
0
Comment
Question by:judico
  • 12
  • 7
19 Comments
 
LVL 19

Accepted Solution

by:
drichards earned 2000 total points
ID: 12389942
Do you know what the error is (partial reads, dropped bytes, incorrect bytes, etc.)?  Hyperterminal just spits all the bytes onto the screen so framing doesn't matter.  You are receiving into a buffer and assuming that each buffer is a complete data sample when in fact it may be a partial sample or have pieces of multiple samples in it.
0
 

Author Comment

by:judico
ID: 12390917
I don’t know what the type of the error is. The only way I know a point is in error is by observing that it appears way off the normal run of the curve (or is outside the bounds of the graph.) Is there a way to determine what kind of error it actually is?

How can the integrity of each byte received by the program be evaluated so that an eventual incorrect byte be ignored?

P.S. I should mention that using different RS232-USB cables (cables from different companies) cause the appearance of different errors (both in number and in value.)
0
 
LVL 19

Expert Comment

by:drichards
ID: 12391089
One way to diagnose errors is to add some code to save the data into a file.  Save the sample number, the number of bytes and the data bytes for each buffer and add a delimiter like CRLF between buffers.  Then examine it at the points where failure occurs and see if you can tell what went wrong.  If you can figure out what the errors are, you can either correct them or filter them out.
0
Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 

Author Comment

by:judico
ID: 12391192
I placed the following code in my program and started observing the raw data sent each time the device is called:

FontStyle style = FontStyle::Bold;
Drawing::Font *arial = new Drawing::Font( S"Arial", 12, style );
graphicsObject->DrawString( String::Concat( S"Raw Data = ", dMeterReadingString), arial, brush, 450, 160 );

The first time the data obtained from the device is always in error (I am filtering this anyway.)

The next call to the device (my first point on the graph) is usually correct. However, instead of obtaining a string, say, “5103” I get a string like “51035106” i.e. consisting of 8 instead of four numbers. In the program I extract the Substring(0,4) so I wouldn’t know that 8 numbers are received and the data appears as correct on the graph.

Second and third point appear also correct (although 8 instead of 4 numbers are received.)

But when, say, fourth point is received the data (the first four symbols) are not numbers any more. Obviously some incorrect read has occurred for some reason.

Then fifth point is again all right (8 numbers.)

And so on ...

Occasionally, in the course of the acquisition several other incorrect reads (non-numerical symbols) occur here and there.
0
 

Author Comment

by:judico
ID: 12392139
The things were greatly improved by changing

config.ByteSize = 7;

to

config.ByteSize = 8;

Now the non-numerical characters disappeared. Nevertheless, faulty data continued to occur. Then I changed the timeout constant from 2000 to your original value:

comTimeOut.ReadTotalTimeoutConstant = 2500;

and the performance was still further improved – only very rarely I would see faulty data. Again, reproducibly, at that (say at 89th point.)

With these latest changes the raw data started arriving, for some reason, as 8 numerical characters (each odd point) and 12 characters (each even point). The first data value (which I ignore anyway) is still in error. Of course, since, as I said before, I take the Substring(0,4), the length of the arriving string doesn’t actually matter.

The still remaining occasional erratic data still bother me, however. As a palliative measure I can include code to filter out the data which are out of bounds. However, erroneous points which are less than the upper and greater than the lower y-axis value are impossible to correct. Therefore, I need to ensure somehow correct acquisition of every single point. What else do you think should be done to achieve that?
0
 

Author Comment

by:judico
ID: 12392179
What I said in the previous posting applies to one of the RS232-USB cables I am using. When I tried the above changes when using another RS232-USB cable (produced by another company) no acquisition of  any data was possible at all (probably the change from byte size 7 to 8 did it.)  Is it that different cables use different protocols of transfer? If so, this would mean that I can use my device with only one particular type of cable. What do you think?
0
 
LVL 19

Expert Comment

by:drichards
ID: 12394990
>> I get a string like “51035106”
I suspect you are just getting two samples at once.  Unless you are exactly synchronized with the meter, you will occassionally get two data points.  Because of your read timeout, you will never get 0 data points.  You should just read 4 characters at a time out of the buffer.  It would also be interesting to see the all the data starting one or two samples before the bad point(s) and ending one or two samples after.

As for the cables, I have no idea as I have never used a USB/serial cable.  Is there any documentation with them?
0
 

Author Comment

by:judico
ID: 12395332
This is really strange. Experiments with three different RS232-USB cables give different results. In fact two of the cables seem to work the same while the third cable behaves completely differently. The manual that comes with it only has instruction as to how to install the cable and no further documentation is available.

I'd like to mention something else. The problems with the faulty points started when I began to look for ways to increase the accuracy. My PIC is 12F675 and is 10 bit which wasn't enough. Therefore, I decided to apply oversampling and averaging with the aim to increase the accuracy. Indeed, the accuracy was greatly increased (about 4 times).

Here is the original program of the PIC which works fine (no faulty points):

' PicBasic Pro program to display 4-channel 10-bit
' A/D conversion on serial terminal
@ DEVICE MCLR_OFF, INTRC_OSC_NOCLKOUT, WDT_OFF, BOD_OFF, PWRT_ON, PROTECT_ON
DEFINE OSCCAL_1K 1 ' Calibrate internal oscillator
DEFINE ADC_BITS 10 ' Set number of bits in result
DEFINE ADC_CLOCK 3 ' Set clock source (3=rc)
DEFINE ADC_SAMPLEUS 100 ' Set sampling time in US
N2400 CON 16780
AD VAR WORD[4] ' Create adval to store result
X VAR BYTE ' Holds each A/D channel result
TRISIO = %11011111
ANSEL = %01111111 ' All A/D Frc osc
CMCON = 7 ' Analog comparators off
ADCON0.7 = 1 ' Right justify
PAUSE 500 ' Wait 500ms to initialize
MAIN :
FOR X = 0 TO 3 ' 4-channel loop
ADCIN X,AD[X] ' Read A/D input
WHILE ADCON0.1 = 1 ' Wait until conversion complete
WEND
@ NOP
NEXT X ' CLose loop
PAUSE 250 ' Pause 250mS
FOR X = 0 TO 3 ' Get 4-channel results
AD[X] = AD[X] & $03FF ' Mask out upper 6-bits of 10-bit results
SEROUT2 GPIO.5,N2400, [DEC X," ", DEC AD[X] ,13,10]
NEXT X
GOTO MAIN ' Loop
END ' END Program

NOTE:the symbol @ precedes code in assembly language

However, as I said it doesn't provide enough accuracy.

To increase accuracy I rewrote the above code whereby I made the ADC take 16 samples which were added to the variable TOTAL whose value is then transferred to the VC++ program where it is divided by 16 to get the average value of the readings. The new PIC program looks like this:

' PicBasic Pro program to display 4-channel 10-bit
' A/D conversion on serial terminal
@ DEVICE MCLR_OFF, INTRC_OSC_NOCLKOUT, WDT_OFF, BOD_OFF, PWRT_ON, PROTECT_ON
DEFINE OSCCAL_1K 1 ' Calibrate internal oscillator
DEFINE ADC_BITS 11 ' Set number of bits in result
'DEFINE ADC_CLOCK 3 ' Set clock source (3=rc)
DEFINE ADC_CLOCK 30 ' Set clock source (3=rc)
DEFINE ADC_SAMPLEUS 100 ' Set sampling time in US
N2400 CON 16780
X VAR BYTE '  Declares variable X
AD VAR WORD[1] ' Create adval to store result -- HEX number
TOTAL VAR word ' Declares valriable TOTAL -- HEX number
TRISIO = %11011111
ANSEL = %01111111 ' All A/D Frc osc
CMCON = 7 ' Analog comparators off
ADCON0.7 = 1 ' Right justify
PAUSE 500 ' Wait 500ms second to initialize

MAIN :

TOTAL = $0
for X = 1 to 16
'PAUSE 45
ADCIN 0,AD[0] ' Read A/D input
TOTAL = TOTAL + AD[0]
@NOP
next X

PAUSE 1000 ' Pause 1000mS
'AD[0] = AD[0] & $07FF ' Mask out upper 6-bits of 10-bit results
SEROUT2 GPIO.5, N2400, [DEC TOTAL]

GOTO MAIN ' Loop
END ' END Program

Most of the time the above program works fine except for these occasional faulty points. I think you are quite right when emphasizing the importance of synchronization between the meter and the VC++ program but so far I'm unable to figure out how to accomplish thas in the present configuration, i.e. when the meter is pumping data continuously at its own pace. One way to deal with the problem, it seems, is to make the meter send data only when called from the VC++ program while being silent the rest of the time. However, I don't know how to do that, especially on the PIC side.
0
 

Author Comment

by:judico
ID: 12396114
Let me mention this too. The problematic RS232-USB cable works differently on different computers all of which are under Windows XP. I never expected there would be such difference among these cables. Something really radical has to be done to ensure proper acquisition of the data.
0
 
LVL 19

Expert Comment

by:drichards
ID: 12396962
Seeing only the bits and pieces it's hard to tell exactly what's going on, but a few observations:

1) The first version of the program reads 4 channels and outputs each one while the second version reads channel 0 16 times and outputs the sum.  How were you reading the results of the forst program?

2) In the first program you were masking each number with 0x03ff.  Don't you need to do this to each sample before adding it to TOTAL in the second program?  If the higher bits are set it would throw off the result.

3) What does [DEC ...] do?  Does this convert the number to a string?

There are a coule of ways to synchronize the reads with the data.  It depends on how the data is being produced.  Is the data just arriving at some frequency and is the number of data bytes received always the same per sample?
0
 

Author Comment

by:judico
ID: 12397991
Firstly, I’d like to show you what palliative solution I found which seems to work so far on my computer with both types of cables. Then I’ll try to answer your questions.

As I mentioned before, the correct data which VC++ code receives when the timeout is

comTimeOut.ReadTotalTimeoutConstant = 2000;

is always 8 numerical characters in length. In passing I should mention that with one of my RS232-USB cables the all-numerical output is achieved by using config.ByteSize = 8; while the other cable for some reason needs config.ByteSize = 7;  ... Anyway ...

Knowing this (i.e. the fact that the output must always be 8 characters long in order to be correct) I placed the following selection statement:

int lenn;
if ( evtCounter > 1)
       {
lenn =  dMeterReadingString->get_Length();


if ( lenn != 8 )
{
TEMPERATUREscaled[ evtCounter ] = TEMPERATUREscaled[ evtCounter - 1 ];
TEMPERATURE[ evtCounter ] = TEMPERATURE[ evtCounter - 1 ];
}
else
{
}
       }

where TEMPERATURE[ evtCounter ] is the array I plot.

With this simple code I seem to be able to achieve a contiguous curve. Of course, it will be desirable to avoid such palliative solutions and guarantee that every point the program receives is a true experimental point corresponding to the given moment of time. For the purposes I will be using this device now, however, the above solution is satisfactory. However, for more precise measurements something has to be done to ensure proper integrity of the data.

Concerning your questions:

1) The results in my first program are read as follows:

dMeterReadingString = mmeter->ReadString();
firstChannelSTRING = dMeterReadingString->Substring(2, 5);
                              
System::String *st = firstChannelSTRING;           
         try
         {         
         dMeterReading = System::Double::Parse(st);
         }
         catch (Exception *exception)
         {
         }

Notice that conversion from string into double can only be done through this exception. Otherwise it doesn’t work. Also, as you can see, i’m reading only the first channel despite the fact that the device sends data from the four channels.

The results in my second program are read in a similar fashion, the only difference being in the Substring due to the difference in the format of the data sent by the device (and, of course, the division by 16):

dMeterReadingString = mmeter->ReadString();
firstChannelSTRING = dMeterReadingString->Substring(0, 4);
                   
System::String *st = firstChannelSTRING;             
         try
         {         
         dMeterReading = System::Double::Parse(st);
         }
         catch (Exception *exception)
         {
         }
dMeterReading = dMeterReading / 16;

2) The use of masking here is unclear to me. Logical adding of 0x03ff contributes nothing, as far as I can see, because it returns the initial values of the bits. Thus, I don’t see what the reason for using this mask is. For the sake of a trial I used it in my second program when I did my latest experiments and, as expected, it made no difference.

3) DEC, as far as I understand, directs the device to output decimal digits through the serial port. These digits are received in the VC++ program by the MultiMeter.cpp placing it in a buffer and from there are used further as a string. This is how I understand it which may not be correct.

As far as the synchronization you mention in your last paragraph, I don’t know how else to use the fact that the correct data arrives in 8-digit packets but by writing the selection statement code I started this posting with.
0
 

Author Comment

by:judico
ID: 12405849
Thanks for the help. The problem is partially solved for now but I need to discuss it further so that a reliable acquisition of all data is achieved without resorting to palliative solutions. I may post a follow-up question to that effect in the future.
0
 
LVL 19

Expert Comment

by:drichards
ID: 12406257
If the good samples have 8 bytes of data, what do the bad ones have, and what is the buffer content at that time?  That may tell you something about the nature of the errors.

As far as the masking, it is to ensure that only the 10 data bits are used.  If there are values in the other bits, they are not part of the 10-bit value.  It would be interesting to see if the 6 non-value bits are ever set.

It would also be a good experiment to just read the serial port as fast as you can an see how the data arrives.  Write a separate program that just reads the port constantly (no waiting) and see how the data comes in.  You can timestamp buffers and then look at how many charactrs arrive at a time and how they are spaced.  This is not entirely accurate because of the way Windows runs the port, but it'll be close enough for this.
0
 

Author Comment

by:judico
ID: 12407360
The number of bytes for the bad data usually differs each time. Also, with different cables I get different type of bad data. It seems to me that the bad data is due to partial reading of the word sent by the device.

One set of data sent by the device (one word) consists of 4 bytes of data. This can be seen using the Hyperterminal. It appears that the VC++ program stays open enough time to allow two words from the device to fill its buffer.

Unfortunately, since, obviously, the cycles of the VC++ program and the PIC program differ there are moments when the buffer of the VC++ program does not manage to acquire the entire two-word set sent by the device and that is when the data in the buffer is bad. This is what the explanation seems to me to be so far. I thought of finding ways to unify the cycles of the VC++ and the PIC programs so that their synchronous running would avoid the partial transfer of data. But then I realized that such sufficient synchronization is probably hardly achievable especially on the side of the VC++ program because of the many factors influencing it while running under Windows. So far, it seems the best approach is the check for integrity of the acquired data which I mentioned in the previous posting. It seems to work quite well for the purposes I intend to use the program at present.

As far as the masking goes, I understand that  $03FF (which is 1111111111) is to ensure that only 10 bits are used. However, there are no more than 10 bits (11 bits in the second program; the mask for them should be $07FF) at a time that are acquired anyway. Say, the 10 bits acquired are 1000101110 and they are to be masked with 1111111111. The result is 1000101110. Thus, nothing is accomplished by this masking. I don’t know what the meaning of the comment “Mask out upper 6-bits of 10-bit results” is. I should have deleted it when I posted the code above.

As for your suggestion to write a separate program to see how the data from the device arrive, I thought the Hyperterminal accomplishes this task. It appears that the data shown on the screen of the Hyperterminal arrives exactly at the pace and the format as the device is sending it – as packets of 4 data bytes each time the device fires.
0
 
LVL 19

Expert Comment

by:drichards
ID: 12407546
>>  I thought the Hyperterminal accomplishes this task
Sort of, but it doesn't give you any timing except whatever you observer which is not very precise.  

Is it possible to change your PIC program to put a marker byte or bytes between data values (like CRLF)?  Then you could easily know where data begins and ends.
0
 

Author Comment

by:judico
ID: 12407788
Yes, I did that. I put CRLF (13, 10):

SEROUT2 GPIO.5, N2400, [DEC TOTAL, 13, 10]

and this is how I ascertained that the word sent by the device consists of 4 data bytes.
0
 

Author Comment

by:judico
ID: 12407800
One thing I still can't understand is why the first ever data packet (which I, of course, ignore) always consists of 5 non-numerical characters first of which is always a small square while the rest of the characters differ every time the program is started.
0
 
LVL 19

Expert Comment

by:drichards
ID: 12407948
If you have the CRLF characters in the data stream, it should be very easy to frame the data.  Just look for data between the CRLF markers.  You will have to buffer data between reads if you get partial values during a read.  As for the seemingly spurios data when you start your program, does your device output some sort of initialization information when the connection is established?  It seems odd that you would get random data on the line at startup.  My first guess would be that the device is doing it on purpose.
0
 

Author Comment

by:judico
ID: 12408221
>> ... it should be very easy to frame the data ...

This is very interesting and could be the solution to the problem. Could you suggest some code to do that? Should it be done in the MultiMeter.cpp or in another part of the program?

As for the initialization, I can see the initialization part in both versions of the PIC program but I don't see how that gets to the output. Could it be that some buffer in the PIC contains residual data and is not flushed before the startup?
0

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The following diagram presents a diamond class hierarchy: As depicted, diamond inheritance denotes when two classes (e.g., CDerived1 and CDerived2), separately extending a common base class (e.g., CBase), are sub classed simultaneously by a fourt…
In Easy String Encryption Using CryptoAPI in C++ (http://www.experts-exchange.com/viewArticle.jsp?aid=1193) I described how to encrypt text and recommended that the encrypted text be stored as a series of hexadecimal digits -- because cyphertext may…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…
Are you ready to place your question in front of subject-matter experts for more timely responses? With the release of Priority Question, Premium Members, Team Accounts and Qualified Experts can now identify the emergent level of their issue, signal…

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question