Geographic Variation of Mobile Application Usability
Department of Computer Science
George Mason University
Department of Computer Science
Grambling State University
Department of Computer Science
Montana State University
Abstract — Rural areas lack connectivity to reliable, cellular data services. Inadequate connectivity delays traffic of mobile cloud-based applications and may impair their usability. However, it is unclear to what degree geographic differences of net- work conditions exist and whether these differences meaningfully degrade cloud-based application performance in rural areas. We propose to measure user requests’ network traffic delays using a client-server application. Network metrics of latency, throughput, and jitter will be colleated to characterize user request delays in different geographic areas on different devices and demonstrate geographic variation of network performance. We will correlate collected measurements with previous research, which has linked request delays to loss of usability . Our results will show to what degree of cloud application network performance disparity translates into reduced usability that inhibits rural users’ participation in modern society.
Keywords:mobile usability, geography, context, empirical study, rural access
With the expansion of the latest 3G and 4G technologies, the availability and demand for cellular data services has increased. However, despite cellular providers' claims of nationwide coverage, many rural users still lack comparable network quality to urban communities. In turn, inadequate cellular internet impairs usability of cloud-based applications and limits rural users’ access to mobile services. Without comparable cellular Internet quality to urban users, rural users may see a digital divide that will inhibit them from mobile access to social media, web browsing, gaming, and video chat.
Unreliable connectivity has led to delayed traffic, which in turn impairs the usability of cloud-based applications. The degrading effect of network delays and losses on data services has already been studied . Yet the degree of disparity between urban and rural connectivity is not yet understood and requires measuring network QoS in those two geographic areas. Speed and reliability of a connection can be inferred through network centric metrics such as delay, path latency, packet loss, and bandwidth.
In order to demonstrate whether geographic differences affect application performance, a client-server application has been implemented. This client-server application serves as a tool for measuring user requests’ network traffic delays. Deploying this network metrics application on various devices and in distinct areas will characterize geographic variation of network performance.
Moreover, our collected measurements expand previous research, such as MIST’s platform . Our results will show to what degree network performance affects usability. Results that show network performance can then relate to application usability through others’ results. It will show why application usability is reduced, which inhibits equal participation for rural users in modern society.
The remainder of this paper is organized as follows. In Section 2 we review background and related work of network metrics. Section 3 exhibits the client-server application architecture. Section 4 will provide an analysis of data collected. Section 5 describes future work. Lastly, we conclude in Section 6 with a summary of our findings.
II. Background and Related Work
Geographic network variation can reduce application usability enough that it leaves behind the rural user population. Restricted usability of mobile applications may be caused by network delays. The research of Luo et. al’s shows that QoS for FTP service can be investigated by measuring delay, throughput and usability. User-perceived quality measurement for a more resource demanding service, such as VoIP, also involves network-centric metrics . Tsetsgee and Lkhagvasuren assess the effect of delays on subjective and objective VoIP quality scores . Kim et al. also investigates application-level performance of voice as well as of data services in mobile networks by considering end-to-end delays and traffic . Knowing that decreased network quality can degrade application performance, our measurement tool focuses only on measuring end-to-end network delays in order to geographically model network quality.
The metrics collected can show the affected behavior of different lower-level protocols on application performance rather than the protocols themselves. Our framework aims to measure this network variation through user-perceived metrics including latency, loss, throughput, and jitter. Other studies similarly focus on the user experience. Mobile Internet Services Test (MIST) measures delivered performance of network quality regardless of various underlying technologies among different providers . Kostanic et al. also focuses on cross-provider, technology-independent QoS assessment with their proposed methodology for comparing cellular networks . In measuring the feasibility of bandwidth estimation in wireless networks, Koutsonikolas and Hu use the achievable throughput as most important bandwidth metric because it represents a single client’s perspective .
While many experiments are performed under laboratory settings, previously listed works are carried out in operational networks. Liu’s research especially addresses how certain wireless technologies compare in performance in both controlled and deployed environments . Another study which evaluates smartphone TCP traffic shows high delays and losses for end-users. These studies show that while some technologies operate well in laboratory settings, the rapidly changing context in the wild provides a more accurate picture of a protocol’s behavior in its actual environment. Testing operational, mobile networks also involves measuring at different times and locations from a base station in order to give a more comprehensive picture of network quality. Likewise, our measurements were completed in varying conditions.
The MIST platform serves as a model for what will be presented in this paper. It measures network metrics, such as latency and throughput. MIST provides information about cellular data networks to aid mobile application development. This paper will further MIST’s research by measuring network metrics in geographic areas. Although our system is modeled after MIST, it also uses a cloud-server that divides processing load over multiple servers and minimizes delays caused by the client’s route to a server . Our system also aims to define the extent of geographic differences in network conditions and whether these differences degrade application performance in rural areas.
Luo et. al constructs a set of paramaters to define QoS for an FTP process which includes delay and throughput. Measurements using their network centric QoS definition agrees with user survey results, demonstrating that usability for an internet service depends on network quality. For example, user-perceived file transferred time is equal to the difference between connection initiation and termination, similar to the time it takes to send and receive a packet. Of the three servers tested, the target server with the smallest delays also received the highest usability results through a user survey .
Measurements of network performance and user behavior were collected at an ACM conference by Balachandran et al. Metrics for user behavior include number of handoffs, types of traffic classes used, user session load and duration as well as packet error, retransmissions, and bandwidth for network metrics. Their results were used to generally characterize user behavior and analyze how workload should be distributed across access points . Rather than a WiFi network and any type of network-capable device, our study focuses on cellular networks and mobile devices.
III. System Architecture
Our network metrics tool measures latency, throughput, and packet loss using a client-server application. The client application can be installed on any Android device running at least API version 15. Meanwhile, the server-side software is hosted in the cloud and can be executed from a laptop computer. Figure 1 illustrates the high-level methodology of the client- server application. The size and number of packets to send may be adjusted from the server and then sent to the client. The client sends and receives first UDP, then TCP packets as specified. Afterwards it sends counters and timestamps back to the server in order to perform final calculations.
A. Packet Loss
One parameter includes the number of UDP packets the client should send. At the server, it counts how many packets are received until a timeout occurs. The server finds the difference in percentage between packets sent and packets received to measure packet loss.
Previous research shows that the changeability of mobile connectivity caused by throughput fluctuations or interference prevents accurate bandwidth estimations . Therefore, we chose to measure TCP throughput directly by measuring the rate at which data travels from one link to another, specifically between the client and server. Timestamps for when packets are sent and received are collected and compared to get a time difference. The aggregate size of sent packets is divided by the time it took the packets to traverse up or down the network in order to compute uplink and downlink speeds.
C. Path Latency
Other information sent to the server includes collections of timestamps. Timestamps are collected from the client’s send and server’s receive times as well as the server’s send and client’s receive times. These values are compared to compute latencies. Uplink latency is measured as the time it takes for a UDP packet to travel from the client to the server. Downlink latency measures the time from the server to the client. Timestamps on both sides are compared to create lists of uplink and downlink latencies while the maximum, minimum, and average are computed for both directions. The same lists of latencies are used to determine uplink and downlink jitter, the difference between the highest and lowest latency values.
Because the system clock on mobile devices may vary, it is necessary to synchronize times between the client and server. We use the same synchronization method used in MIST. It requires four timestamps for two nodes, client C and Server S: when C sends, S receives, S responds, and C receives response. The offset is calculated by: and used to adjust server-side times . This allows us to calculate accurate latency values between the client and server.
An analysis of a small data set collected at Montana State University’s campus is demonstrated to show the effectiveness of the client-server application. Our primary goal is to identify network causes of degraded application performance. Our second goal is to characterize geographic differences in network performance of mobile application usability.
A. Data Collection
The data in the following sections were gathered from a Motorola Droid Razr cellular phone on the Verizon Wireless network. Tests were run from a stationary location in Verizon’s CDMA network using either 3G or 4G LTE communication protocol. The three areas from which measurements were collected were from the Strand Union Building (SUB) and a dormitory that provides 4G mobile Internet and from an office indoors limited to a 3G connection. The tests were also completed at two different times: between 10AM-11AM and between 5PM-6PM. The application was installed from a laptop computer to the Android phone. For each set of measurements, the cloud-hosted server was first executed from a laptop. Then the client application requested a connection to start the measurement process. Following the series of send- receive packets, the server collected and printed results to the laptop console.
Metrics collected include: throughput, path latency, and packet loss. We measured throughput directly using a bulk data transfer rather than through packet pair or other estimation techniques. Parameters for throughput measurements involved 200 TCP packets of a small size, 100 bytes. For the following latency analyses, we sent and received 200 UDP packets of size 1400 bytes. The same series of packet messages was used to measure packet loss.
B. Latency Analysis
For each of the 200 packet transmissions, timestamps were saved when sent and received on each side. Uplink and down- link latency measurements are a major part of our framework because the speed at which information traverses the network impacts the speed of application performance.
Figure 2 describes downlink latencies for measurements taken in the afternoon. Downlink latency in a 3G location varies from 53 ms to 508 ms with an average of 85 ms. Studies done by Tsetsgee and Lkhagvasurem show that delays below 10 ms do not affect perceived quality for VoIP and is still acceptable up to 250 ms. The average latency measured allows for a high quality VoIP conversation. However, the maximum delay exceeds the 250 ms threshold for acceptable conversation quality. The same study also specifies jitter guidelines for VoIP quality. The 455 ms jitter is over 75 ms, which would cause too much jumble in the conversation. On the other hand, jitter in a 4G network was only 15 ms and jitter below 40 ms does not affect conversation . The same set of 4G measurements shows acceptable downlink latency for VoIP. The latency ranged from 46-61 ms and is below 100 ms, so it does not cause a detectable delay.
Figure 3 shows uplink latency measured in the morning. The range of latency values in a 3G area vary from 50 ms to 239 ms. The work of Kim et al. show that the delay requirements for conversation, streaming, and interactive applications are 100ms, 250ms, and 400ms, respectively . The average uplink at 72 ms means the network quality is adequate for all previously listed applications. However, if the latency were to remain close to the maximum at 239 ms, conversation applications such as VoIP would not be usable. Uplink jitter in a 3G network was 189 ms. Like the morning measurements, jitter was above 75 ms making VoIP unusable. Similar to afternoon tests, only a 15 ms jitter was found in the 4G network. This also falls below 40 ms, making jitter undetectable. Meanwhile the uplink latency for a 4G network measured in the same time frame varies from 45 ms to 59 ms. Even the maximum delay in this location fulfills Kim’s listed delay requirements, meaning the network can support all types of applications .
C. Throughput Analysis
Throughput analysis is considered between the 3G and 4G areas. TCP throughput may sacrifice time for its reliable transmission protocol. We use the two datasets to compare how throughput is affected by connection level in Figure 4.
Network performance measurements demonstrate that different workloads operate at different throughput data rates. In Balachandran et al’s measurement study, a light workload type used less than 15 Kbps, a medium one used 15-80 Kbps, and heavy sessions ran at over 80 Kbps . According to their results, our downlink throughput measurements of 277 Kbps and 250 Kbps in 4G and 3G networks would work well for any type of session load. However, an interactive application which may make more use of uplink data transfers would not operate as well. Uplink throughput in a 4G network was limited to 50 Kbps and 48 Kbps in a 3G location. This speed would only allow for light to medium workload processing.
D. Packet Loss Analysis
For every data set collected, all packets were received and packet loss stayed at 0 percent. However, rather than an optimistic picture of the network, these values actually revealed a glitch in our application. Packet loss was determined by comparing the number of packets received at the server to packets actually sent from the client. Yet if less than the specified number of packets were received, neither side would continue to collect metrics. Instead, subsequent measurements and those that required a TCP connection were disregarded.
V. Future Work
Besides packet loss, data collection was generally challenging. After implementation, initial software testing was executed over a Wi-Fi network to a server hosted on a laptop. However, limitations caused by the network’s firewall prevented the server from receiving UDP packets. Even when the server was moved onto the cloud, limitations caused by network connection timeouts meant only a limited amount of tests could be fully executed.
We were only able to perform tests on one mobile device and two locations. The tests were performed on the Verizon Wireless network on an Android 4.0.3 platform. Nevertheless, in order for our test to be thorough we will need to perform tests on more than one cellular device, cellular network, and in various geographic areas, apart from the Montana State University campus. To expand our research, users with cellular devices in widespread locations will be able to download the application and participate in our data collection efforts. We will then be able to accurately measure geographic differences in mobile application usability in rural and urban areas. Measurements should also be run for each location throughout the day in order to test under different traffic. This involves either measuring data at consistent time intervals in varied set locations or at different time periods through the day.
Moreover, we expect to measure additional network performance factors that impact the performance of application traffic. Though latency, throughput, packet loss, and jitter give us an overview of the geographic variation of mobile application usability, other network metrics will provide more information. Furthering our endeavors involves measuring the changes in overhead sizes caused by different packet lengths, variations in latencies for different priority packets, and timing and queuing delays of different packet scheduling schemes.
We have presented an end-user focused network metrics tool which operates on a mobile platform and communicates with a cloud server in order to measure network performance. Sample data was collected and analyzed to show the large variability of delay, loss, and bandwidth in different networks. This measurement framework
aims to characterize geographic differences in mobile network quality. Future tests using this tool can show the impact of delays on cloud-application performance in order to illustrate the degree of digital divide between urban and rural communities.
 Balachandran, A.; Voelker, G. M.; Bahl, P. and Rangan, P. V. “Characterizing user behavior and network performance in a public wireless LAN” Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems 2002
 Tsetsgee, B. and Lkhagvasuren, T. “The Study On Quality Requirements of Interactive Voice in IP Network” International Forum on Strategic Technologies June 2008
 Kim, J.-H.; Lee, H.-J.; Oh, S.-M. and Cho, S.-H. “Performance Modeling and Evaluation of Data/Voice Services in Wireless Networks” Vol. 14, 233-246, March 2008
 Wittie, M. P.; Stone-Gross, B.; Almeroth, K. C. and Belding, E. M. “MIST: Cellular Data Network Measurement for Mobile Applications Broadband Communications, Networks and Systems” BROADNETS. International Conference on Broadband Communications, Networks and Systems September 2007.
 Falaki, H.; Lymberopoulos, D.; Mahajan, R.; Kandula, S. and Estrin, D. “A First Look At Traffic On Smartphones” Proceedings of the ACM SIGCOMM Conference on Internet Measurement 2010
 Liu, X.; Sridharan, A.; Machiraju, S.; Seshadri, M. and Zang, H. “Experiences in a 3G Network: Interplay between the Wireless Channel and Applications” Proceedings of the ACM International Conference on Mobile Computing and Networking 2008
 Koutsonikolas, D. and Hu, Y. C. “On the feasibility of bandwidth estimation in wireless access networks” Wirel. Netw. Vol. 17, 1561-1580, August 2011
 Kostanic, I.; Mijatovic, N. and Vest, S. “Measurement Based QoS Comparison of Cellular Communication Networks” IEEE International Workshop Technical Committee on Communications Quality and Reliability May 2009
 Mills, D. L. “Improved algorithms for synchronizing computer network clocks” Proceedings of the Conference on Communications Architectures, Protocols and Applications 1994
 Luo, Z.; Liu, F. and Xie, Y. “User-perceived FTP service QoS parameters and measurement” IEEE Conference on Network Infrastructure and Digital Content November 2009