Ougoing SMS collison problem (possible 3582 pts!)

I'll do better than give you source code, I'll give you an opensource high volume server ...

If you aren't too attached to your code, then, I would suggest using an open source gateway such as Kannel to get the job done.  Kannel in use by very high volume environments and I do believe that you can do load-balancing with it as well.  You can also directly connect this to a number of SMSC's once you have it stablized for you envirnment.  You can download Kannel from kannel.3glab.org.

Based on the VB comments, the only down side for you may be that it is a Linux-based server.  You'll need Cygwin to run it under windows.

If you have this many messages then you should be seriously considering an SMS Gateway (like Kannel) so that you can at least connect directly to the carrier's SMSC to save on incoming request costs and to handle volume.

---

Now regarding your current problem.  I don't believe that collisions can cause you to receive half-SMS's - may be an issue with Wavecom modems.  The SMSC will either fully deliver the message to your module based on module ack or it will store it in a store-and-forward queue for later delivery.  This sounds like it could be module memory issue, or load on the wavecom processor issue.  

- Are there any messages stored on the modem?
- Are you polling the modem (not good) or using the indicator events like CIND.
- Also, I don't believe that this should happen, but who knows in a high volume environment.  Could the incoming +CIND interrupts be cutting into your parsing algorithm - that is, are you sure (via logging) that the messages are in-fact half-messages or could something like a +CIND or +WIND be throwing off the parser - just a thought ...

Mathew

thanks for your suggestions, folks
----------
BigRat.
have been looking at AT+CPAS for a long time, but as you have already mentioned, the delay to wait for the cpas reply still does not guarantee that the instance of sending out is free from collision.

for instance
MSComm1.Output = "AT+CPAS" & Chr$(13) & Chr$(10)

then should i apply a boolean lock and go into a doevents loop? or should i specify a 1 second timeout? i already have several loop cycles waiting for triggers and boolean lock/unlocks, and its hitting the processors pretty bad. sigh. need optimisation somewhere...
----------

mtasalloti.
i looked at kannal when i first started developing the gateway. many of the algorithms in the gateway is derived from kannal's open source. however, trying to puzzle out the language got a bit too much for me, and i could not customize it to bundle into one complete application- gateway solution. kannal uses mutex locks to handle flow control, if i recall, and there was no equivalent in VB. hence the problem arising...

incoming sms Costs are born by requestor. outgoing sms will be marginally cheaper, but not enough to cover capital outlay.


- Are there any messages stored on the modem?
all messages on the modem are cleared before attmepting to send outgoing.

- Are you polling the modem (not good) or using the indicator events like CIND.
periodic polling of modems, and then using MSMQ triggers to reduce secondary polling for processing. how do i use CIND ?

- Also, I don't believe that this should happen, but who knows in a high volume environment.  Could the incoming +CIND interrupts be cutting into your parsing algorithm - that is, are you sure (via logging) that the messages are in-fact half-messages or could something like a +CIND or +WIND be throwing off the parser - just a thought ...

there is no problem of half-received SMSes. its a problem of outgoing sms colliding with half-incoming SMS and causing the application to hang. have not figured out how to trap the error equivalent to "message sending failed". any ideas?


----------

bigrat and mtasallotti...

the setup is as follows - up to 4 gsm modems, with a simcard of a different mobile network in each one. all modems process incoming messages irregardless of source mobile network, but outgoing sms to any particular network is only sent using the simcard-modem pair of that network.  012-> 012 , etc..

each modem monitor runs in a seperate thread. yes, using vb6.  i'll have to look into vb.net's multithreading abilities in the near future, but the problem at hand is most crucial at the moment. multithreads and inter-thread communication has been resolved ages ago. no crashes or interference with operations. (took me a while to get it right)

dedicating a single outgoing modem for outgoing is insufficient. real time delays at queue cause problems for my users and their clients. (its a retail application distributing goods via sms)

as for SMSC connectivity, local telcos require a leased line into their smsc. they do not accept sms-transmission via public internet. capital outlay for laying the infrastructure to even -one- telco will set me back at least 2 years in profit margins. sigh. moreover, i would have to commit to some crazy figure like 2 million smses a month. the problem i face now is not high sms volume, but high sms traffic at peak hours. volume has not reached those levels yet. international resellers cause delays and are prone to downtime or delays. (keeping a client waiting 5 minutes at the counter is NOT a good thing, trust me. *grin*)

hope the further information is useful. thanks again for the comments.

JooZ

Wow, now it gets even more interesting ... sounds like you've working on this for a while.  Thanks for the backround on your project so far.

Ok, I'm grasping a bit since I don't see why modem messages should hang applications (I take it that the app is waiting for something that its not receiving).  

So, what if we take as much load off the modem as possible.  What do you need to poll for at all?  The WIND and CMTI commands will provide you with everything your application needs.

If, you are getting the messages off the modem using CMGR, then, what I would suggest is to modify your code to simply receive new messages as forwarded from the modem using CMTI. Anyways, excuse the CIND thing, that's either and Ericsson or Philips indication message.  The Wavecom equivilant is CMTI and you set it through CNMI.

So with AT+CNMI=2,2,0,1,0  .. your server will just be forwarded with the full incoming message as it arrives through CMTI unsolicited messages.  *** With this CNMI, you'll also turn on +CDS unsolicited messages which will tell you about the status of your sent items. ***

One last item - are the cut off messages particular to a specific mobile operator.  I wonder if there could be something wrong with their SMSC.

Hope this helps

"... local telcos require a leased line into their smsc. they do not accept sms-transmission via public internet. .... moreover, i would have to commit to some crazy figure like 2 million smses a month."

Simply amazing! One can run the protocol through S-Tunnel with certificates. And 2,000,000! Who sends that many?

"...modem hanging the application...."

I do get this problem on Windows with my Nokia phone (6210), but not on Linux with a Siemens TC35.

Question: How long does a message stay in SI storage? Until it is read (read then delete) or until the App has processed it (read, process, delete send).

Question: What speed are you driving the modem?

Hi,

I have developed a SMS system just like you, an SMS Sending and receiving program that can send 900sms/hour over a SIEMENS TC35 Terminal using Visual Basic, this project is for the market so I will not give (I do not have the rights of direct source code because I have developed this project for a company as freelance programmer) the exact source code but I will give detailed steps how-to avoid/minimize the collision especially under VB using mscomm control. I have faced the same issues like as yours. Well, If you do the following arrangements to your program you will be able to send (with a SIEMENS TC35 GSM Modem) about 900 SMS'es per hour while receiving incoming ones and sms status reports simulatneously without any side effects.

Well, there are some rules that you have to follow while dealing with a GSM Modem;

1. After sending a AT Command to device you must wait response from device and then send another one (this is forcing programmer to use multithreaded programmer with a tight "command pooling" algorithm)

2. If an exact command to be send second time (etc. failure of first command, sending a new one may be required of the same command.) make sure to use "A/" which is "Repeat previous command line"

3. Disable all redundant functions (I will explain this later in detailed manner)

4. Make sure that your connection to device is as follows
      a. Open port in 57600 rather than 19200 (Use setting mscomm.settings="57600,N,8,1") (DO NOT Use 115200, it may result unwanted situations even your computer and modem supports it)
      b. Open connection in Hardware flow control mode (mscomm.handshaking=comRTS)
      c. Set Comm Controls' Null Discard to true (mscomm.NullDiscard = True)
      d. Set comm control's (here after will be referred as CC) in/out buffer to 512 or lower (regarding as your needs more little value more good)
      f. Set CC's RTreshold Value = 1 for evaluting output without harrassing CC and Device (This is important and I will mention more about this.)
      g. Set local rate to a fixed value that you have opened port (AT+IPR=57600)
      h. Set FlowControl Mode from Device to Hardware flow Control using "AT\Q3" command
      i. ***** Disable Command Echo by using "ATE0" command this will shutdown the entered commands to redirected to output, as a result of this you will not be able to see what you have entered as Command, but this takes nearly 50% (in some cases 60%, especially when sending SMS!) weight from your GSM modem and enhances performance greatly *****
      The settings found above will ensure that your communication with your GSM Modem at top rate and very healthy.

How-to Send AT Commands to Device (Explanation of Step 1 which is written above);
Golden rule is to wait one commands "full" response from device and send another one after response received; but how ?
As I wrote the whole program under Visual Basic 6 I was unable to multithreaded program; but I came up a pooling like solution for this limitation;
      Create global function in a module name it sth like SendATCommand which only returns Boolean value (True or False) this will act spinal of the squential pooling system (which is somewhat lame but works fine) this function takes arguments of AT Command to be send, expected result and will return that if the command was successful (True) or failed (False)
      Create a Global Variable sth like BufferStatus and make it integer (in our case it will take only three values as 0,1 or 2). This will trigger CheckedDelay function receive buffer from CC!
      Create another function (this may not be a global one which will be used primarily by SendATCommand function to analyze output of device) named as sth like CheckedDelay.
      Optionally (Generally must but not neccessarily if you do not want.) create a sub which will and name it sth EvalOut evaluate output which is other than the commands expected result and will be called within SendAtCmd here are the code samples for these ones!

Function SendATCommand(Cmd As String, Optional DelayTime As Integer, _
                                    Optional ExpectedResult As String,optional termchar as string) As Boolean

        SendATCmd = False ' set to failed status from beginning
       
        If DelayTime = Empty Then DelayTime = 3
        If ExpectedResult = Empty Then ExpectedResult = DefaultResponse ' deafult Response is defined outside of this function and is vbcrlf & "OK" & Vbcrlf
        If TermChar = Empty Then TermChar = DefaultTermChar ' deafult termchar is defined outside of this function and is vbcr
       
        BufferStatus = 2 ' Global: BufferStatus; Ready to accept incoming from Device
       
        CCtl.Output = Cmd & TermChar ' Send command to CC
       
        If CheckedDelay(DelayTime, ExpectedResult) = True Then SendATCmd = True 'if Expected result met from output that this command was successful

End Function

Private Function CDelay(DelayTime As Integer, SrcStr As String) As Boolean
   
    Dim PauseTime As Long
    Dim Start As Long
    Dim Buffer As String
    Dim SBuffer As String
       
        Do Until BufferStatus = 1 ' Wait until CC's onComm Event's RTreshold property sets this to 1 from 2
            DoEvents
        Loop

        BufferStatus = 0 ' When CC's oncomm event sets this to 2 to 1 set 1 to 0 to avoid conflicts until this function executed completely once!
           
        CDelay = False
        PauseTime = DelayTime
        Start = Timer
       
        Do While Timer < Start + PauseTime ' this will ensure if anything missed from CC's oncomm event and will continue to check output until the expected values are reached

            Buffer = CCtl.Input
            SBuffer = SBuffer & Buffer

            If InStr(SBuffer, SrcStr) Then
                CDelay = True
                Exit Function
            End If

            If InStr(SBuffer, DefaultError) Then ' default error is "ERROR"
                CDelay = False
                Exit Function
            End If
           
            DoEvents

        Loop

End Function


by the way do not forget the following about CC;

Private Sub Comm_OnComm()

      Select Case Comm.CommEvent
           
            Case comEvReceive   ' Received RThreshold # of chars. We have set this befare to 1 to catch every character.
                If BufferStatus = 2 Then BufferStatus = 1
       
        End Select

End Sub

and always use Extended error reporting by entering the following command to the Device, this will enable you to see why an error occured;
AT+CMEE=1

this will enable output error's with their result codes such as "+CMS Error : 500" then you will be able to know why an error occured.
Mostly two types of errors will occur while and SMS Exchange with SMSC
1st is error Nr 500 which is "unknown error" : This is the SMS Collision while sending SMS if this occurs! and if in a SMS Send session this occurs you must give a 200ms delay after every AT Command entered to device for the safety after, one command (AT+CMGS for exmaple) is succeeded you may cancel 200ms delay safely!
2nd is error Nr 42: This is refferred as Congestion/Switching Equipment Congestion means that either your cell or destination cell too busy to handle the SMS you are trying to send or the path is closed temporarily.
Solution to 2nd is to send SMS to that destination with a delay (10S to 1m)

A very Important point is to Send SMS'es in PDU mode rather than Text Mode because GSM Modem will better handle pdu mode and will be much much faster because there will be no internal conversion from text-mode to pdu mode! (This is really really important about reliability)
If you are too lazy to write a Text - to - PDU Conversion (I know it is not easy but ... because I have written :) ) here is a quick trick for you ;
Switch Modem to Text Mode using AT+CMGF=1
then Write your SMS to SIM by using AT+CMGW
and then Swicth Modem to PDU Mode using AT+CMGF=0
Now Send your SMS from SIM using AT+CMSS
Afterwards you will have to delete the sent SMS by using AT+CMGD
and if you need do not forget to Swicth Modem to Text Mode using AT+CMGF=1 again.
This works well, but slows the actual progress as you have to do many steps.
So I recommend you write a Text <--> PDU Converter and work in full PDU mode

Another very important point is (as already meantioned here) receive incoming SMS and Reports in Display only mode;
(and evaluate them in a sub while receiving buffer from device)
to do this set
      AT+CSMS=1
      AT+CNMI = 3,2,0,1,1 (Check these values from you modems manual, your modem may not support these, find equivalent values for yuor self)
well now do not forget that you will receive any incoming SMS directly in to CC's buffer NOT TO SIM Card, so as soon as you receive +CMT: ... or +CDS: ...
you have to enter command AT+CNMA (new sms message acknowledge) or otherwise you CNMI setting will be reset and you will no longer receive reports and incoming SMS'es will start to fall SIM Card again.
When you Enter AT+CNMA device will return "OK" only your operator is waiting an Acknowledge from you and not timed out yet. (about 5 seconds)
I know this is a bit obscure but all the way it is.


2. Disabling redundant functions;
      if you do not need;
            a. Disable Cell Broadcast Messages immediately and never receive them
            b. Disable all functionality except SMS by using Barring function with AT+CLCK="AI",1,0000,247,1 and AT+CLCK="AO",1,0000,247,1

            these will greatly enhance modem's responsiveness

Some last things to say,
      avoid mutliple cycles, use only one if you can (Define a HQ and let it control all the things squentially, one by one)
      DoEvents is a Gift from heaven to VB programmers but it costs too much CPU Power and in some such rare cases may screw up things so use it wisely and not all the time.
      lastly and if you feel that you need to give any delay somewhere you can use 100ms (if you are a paranoid like me you can use 200ms also :) ) but more is absolutely waste of time.

I hope that I may help you a little bit, you may ask whatever you want about, I will appreciate to help in this topic everyone.

      
Onur Buyukcaglar

Thanks Onur!

reading thru your comment, i believe it may hold the key to my problems. I have to take a bit of time to tear open my project file and make the necessary changes to it and see if that works. If anyone else is wondering what happened to my situation, I moved the server back to my house and work from home now for easy maintenance. *grin*

might as well turn a big problem to my benefit in some way or other. hehehh.

Onur, it seems you have spent at least as much time on this as i me, and gone far more in depth that i ever would. Great stuff man! esp the bits abt text vs PDU speeds and redundant functions. My whole program is full of cycles, unfortunately, but upon checking, most of it was caused by the necessity for modem response. will ty and renovate the flow so that a master controller will handle it.

as for multithreading issues..
i use Srideep Prasad's MThreadVB object. It can be obtained here http://www.planetsourcecode.com/vb/scripts/ShowCode.asp?txtCodeId=27481&lngWId=1

Srideep is an amazing person that has managed to get around the limitations of VB in a funky manner. You might want to try it out. I basically create a MDI Parent, then spawn 4 MThreadVB sessions to handle 2 modems, one listener(parser) module and one Sending module, all with a child output screen each. The ocx which controls each modem is simply encapsulated in a window spawned by the modem threads. in this way. its theoretically possible to have up to 4 modems running from a single gateway app. (obviously an I/O card is required to support the extra two serial port connectors)

at this point tho, i havent even dared to go up to 3 modems, tho my experience with the SMS gateway that i based my program upon handles 3 modems fine. I dont think it was written in VB tho. heh...

I know for a fact that running two modems is fine using  MthreadVB. Theres a memory leak in my program somewhere that requires restarting every now and then (anything from several hours to several days, totally random) but i'm not sure if its caused by the multithreading object or just bad coding somewhere.

And, to cap things off, after over a year of trying to puzzle things out and many sleepless nights, i find that my regional distributor of Wavecom modems has FINALLY released an API, obtainable at http://www.dinkum.com.sg/products/hardware-wmapi.asp

sigh.. i'm still wondering if i should purchase it and be done with it or actually continue trying to fix the bugs in my .ocx

cheers
JooZ

Hi Jooz,

"My whole program is full of cycles, unfortunately, but upon checking, most of it was caused by the necessity for modem response. will ty and renovate the flow so that a master controller will handle it"
Well OK, modifying this will not be a problem because most of the AT Commands (roughly about 85% percent of them) will return either CrLf & "OK" & CrLf or CrLf & "Error" & CrLf (if you do not use Extended error reporting) so it is easy to handle mosta of them due to the tact they will return same things over and over again, well you might need some special responses regarding to the comments responses, I mean that command will return again sth like "OK" or "Error" but before these you may need middle responses. Lets say that you need the Manufacturer ID by using AT+GMI, this will return OK/Error at the end but will also return a middle value such as SIEMENS in my case. so I have to trap this value. Well this situation rather easy when you build you HQ, becasue when your HQ is completed you would be proccessing all most of the AT Commands, and the rest is either you are waiting a custom known or unknown result. You can do this easily by implementing a single "Search within Buffer" function that is fired from your HQ every time while it examines the portion of CC's Buffer.
And another topic that you mention it I have downloaded MThreadVB, it is really astonishing, but, there is an important issue for me. There are ten-thousands of ocxs, components are floating right outside, well whenever I start a program sth I always managed to find either sth similar or a like perfect match components DLLs or even completely free programs, well at the beginning I was using them happily which they were helping greatly, but I see soon that there is no challenge in it, I mean that I am programming sth, but the heart of the program was from another guy. did "I" code this program ? Well as I said there is no challenge in it. hehe!
As for your situation, try to estimate a timeline JOOz, If you do not have a time obligation/deadline you better renovate your code and drop ocx from my point of view, because thiss will increase your manageability over your own code and on the other side if that ocx has some bugs and flaws it will be hard to pretend someone else's code and to correct them.
by the way there is an issue about GSM Modems, SIEMENS TC35 supports multiplex mode (AT+CMUX), iy your modem supports it, I may give some basic info about this mode, now I am currently working on this mode because when using this mode you may be able to double-or-triple you application and modem performance.

Thats all for now.

Onur Buyukcaglar

Hi,

Can anybody help me to develope a gateway using VB. I were asked  to develope a SMS Gateway using VB. So that I am trying to find some model source code.

Please send me if you dont mind to study a model of SMS gateway which created using VB.
My Email is jklanka@hotmail.com / jklanka@yahoo.com

Thanks

JK

first of all, many apologies to the xperts who have helped me and to experts-exchange for neglecting my duties towards my fellow experts who have helped me thus far.

i have stopped development on the GSM momdem integration with my appliation partially due to the fact that my company was taken over my dingbats who know nothing about information technology and also partially due to the overload of other real life issues that have consumed my time for the past few months. since then, i have left that company and have been concentrating on other projects such as failover clustering for SQL servers. :p

i am not sure if i wil ever touch the application again due to my inability to obtain investment into capital for a new company that i was planning to start up based upon SMS aggregation (those of you who do not know what aggregation is, feel free to email me and i'll explain to the fullest of my abilities) and i have been disheartened by the whole process to the point that i'm almost ready to turn my back upon SMS technology altogether.

i am 90% sure that any issues of multithreading have been sorted out due to the release of vb.net (correct me if i am wrong), message-collision-handling *should* be handled by the API from Dinkum and that any of you who are just beginning to explore GSM/SMS technology should not have any problems due to the high availability of commercial-grade solutions out there now.

points for this question go to Shadowlight aka Onur for being comprehensive in his explanation of his suggested answer. i am sure that your input has helped many fledgling developers out there.

for the other experts who have suggested answers, many thanks for your input. i am very appreciated for your time and effort in replying to my dilemma, and i wish that i could have put some of it to good use in the end, but ultimately, i never got around to correcting the whole application to the point that i could be proud of it. i am sure some of you know what i've gone thru from past experience. :/

good luck to all of you. be L337 and code like a demon. :) perhaps you might be appreciated more for your skills where you are than where i am.

sincerely,
JooZ