SMS Testing: Text Message Delivery Time and Reliability in Tanzania
September 6, 2011 Editor 0
I am Michael Benedict and while working with SMS-based applications I’ve noticed an air of mystery around the issue of reliability. I hear colleagues say ambiguous things like “carriers consider SMS to be lower priority than voice”, or “SMS delivery isn’t guaranteed”.
My personal experience has been that messages are almost always delivered quickly and correctly, but I’ve heard stories of hours-long delays, corrupt data, and occasionally messages that never arrive. Since I am working on two projects that depend on reliable SMS service — one involves field-based data collection and another employs SMS as a transport layer between computers — I’m curious about how factors such as network, location, and time of day impact message transmission.
I found myself in Mwanza, Tanzania last week with two GPRS modems and a local partner who was unenthusiastic about the work I was actually there to do, so I tried a little experiment. I bought SIM cards and airtime for three of the major TZ networks, put two at a time in the modems, and wrote a simple python script using Adam Mckaig’s excellent pygsm library.
The script sent messages in all permutations at an arbitrarily chosen interval of about 20 minutes. This means that four messages were sent at a time: each SIM to itself and to the other SIM, repeated about every 20 minutes for periods of up to 27 hours. Outgoing messages noted the origin and destination networks, send time, and a sequential message number to make it easier to spot lost messages. Incoming SMS were stamped with the time received, and the delay in seconds between sending and delivery. Messages were sent two seconds apart to reduce errors from the modems trying to send and receive at the same time, and results were recorded in a CSV file.
a) Airtel and Zantel
Zantel is one of the smaller networks in TZ, but I was having technical problems with my Tigo SIM so started with Zantel and Airtel. Airtel was known as Zain until recently when Zain was acquired by Baharti Airtel, based in India. The script ran from about 1AM on January 14 to just before 5AM on January 15. Here are the results:
SMS delivery times between Airtel and Tigo SIMs on January 14, 2011. Click image to see scale. *Messages marked unreadable appear to have reached the modems but could not be read, and resulted in an (unfortunately unlogged) exception in pygsm.
192 messages were sent and 3 (1.6%) were not readable by the modem. Four distinct delivery time ranges emerged. I eliminated one outlier from the plot to make it easier to view: on January 15 at 11:47, a Tigo-Airtel message took 129 seconds to arrive. I believe that the 3 unreadable Tigo-Tigo messages reached the modem, but the script raised an exception and the messages were not read. Unfortunately I was not logging pygsm’s debugging output so I am marking the missing messages as “unreadable” instead of dropped. I will have to log output in future runs and see if I can reproduce the problem. I didn’t expect Tigo-Airtel messages to arrive slower than Airtel-Tigo. The difference is small but it would be interesting to know why it’s there.
c) Two Tigo SIMs.
The purpose of having the modems send messages to themselves in the previous two runs was to test delivery times within a network. It occurred to me that sending from a SIM to itself may not be equivalent to sending between two different SIMs on the same network, so I bought a second Tigo SIM:
SMS delivery times between two Tigo SIMs. Click to scale.
164 messages were sent and 1 (0.6%) was not received. The data shown excludes one outlier: the first Tigo A-Tigo B message took 457 seconds to arrive. So: is SIM to SIM within a network equivalent to sending from a SIM to itself? Sometimes. I don’t know why it varies, but note that the number for Tigo A is +255716379091 while Tigo B is +255717435798.
In particular, after the 255 country code the numbers have different prefixes, 716 for A and 717 for B. Typically Tanzanian SIMs on the same network have the same first 3 digits, and the difference here may indicate that Tigo’s servers treat the two numbers differently. The Airtel/Tigo data above used the SIM that is labeled as “Tigo B” here.
Thoughts and next steps
Unexpected patterns emerged, including asymmetric delivery times between networks, that I would like to understand better. The few dropped messages do not seem to follow a clear pattern, which is a concern for both human and computer-focused SMS applications where missing messages can cause confusion and frustration.
It seems possible that having the modems send all 4 messages with only a 2 second delay between sending could be impacting the results. It would be interesting to revise the script to randomly offset the message send times by a few minutes and see if that increases reliability.
This is not a scientific experiment since there are variables such as time of day, day of the week, network traffic, modem location, weather, and likely many others that are not being controlled for. Still I think the data gives a representative snapshot of network performance. Overall SMS delivery was quite good. With the exception of a few of outliers most messages were delivered in under a minute, and Airtel consistently delivered on-network SMS in less than 10 seconds.
Occasional dropped messages show that safeguards need to be in place if SMS is used for critical applications. Some preliminary timing data I took in Uganda suggests that the networks there are not nearly as reliable. I’m looking forward to looking more carefully at Ugandan networks when I’m back.
If you like this sort of thing and also have too many modems and not enough social life, feel free to try out the script yourself. I’d be interested to see what the results look like in other countries.
This was posted first as SMSpecially Reliable? Part 1 and is repulblished here with permission
Get ICTworks 3x a week – enter your email address:
- Simple technologies still matter
- Did the Kenyan Elections Really Need 3 Maps, 7 Phone Numbers, and Several Web Forms?
- 12 Lessons Learned with ICTs for Monitoring and Accountability
- Google launch SMS version of Gmail in Africa
- Put Your New Business Model to the Test
- Sensors for MERL: What Works? What Does Not? What Have We Learned?
Subscribe to our stories
- In pictures: the 2019 Africa RISING Tanzania monitoring visit August 30, 2019
- Active Internationalization of Software Enterprises: Scale Development and Validation August 30, 2019
- The Manager’s Guide to Leveraging Disruption August 30, 2019
- Key take-aways from a recent Africa RISING exchange visit in Ghana August 30, 2019
- Device that recycles vaporized water from power plants wins MIT $100K May 28, 2019