A comparison of paper-based versus smart phone-based data collection system for field-based survey in two districts of Karnataka

Introduction

One of the goals of field-based survey is to assess the impact of disease by providing useful information to public health authorities.¹ For survey to be effective, data collection systems should be able to maintain data integrity, quickly provide analysis-ready data, and be sustainable to run.² In the last decade, information and communication technology has experienced immense growth and development.³ In developing country like India, data collected using an electronic device has the potential to produce timely and accurate data while reducing expenses on paper, storage space, and entry time.⁴ The tobacco epidemic is one of the biggest public health threats the world has ever faced, killing more than 70 lakhs people a year. More than 60 lakhs of those deaths are the result of direct tobacco use while around 9 lakhs are the result of non-smokers being exposed to second-hand smoke. One person dies every 6 seconds due to tobacco. Up to half of current users will eventually die of a tobacco-related disease.⁵ Around 12% of deaths among adults aged 30 years and over (5% of communicable and 14% of noncommunicable diseases) were attributed to tobacco use. Of the deaths due to non-communicable diseases, 10% of all deaths from cardiovascular system, 22% of cancer deaths and 36% of all deaths from respiratory system are due to tobacco use.⁵

Nearly 80% of the smokers worldwide live in Low - and Middle - Income countries, where the burden of tobacco-related illness and death is heaviest. Tobacco users who die prematurely deprive their families of income, raise the cost of health care and hinder economic development.⁵ The adverse effects of tobacco use and exposure extends well beyond the health risks to individuals. For families, communities and governments, tobacco use and exposure to second hand smoke poses significant social and economic handicap, but also importantly contribute to the major risk factor in the looming epidemic of non-communicable diseases that threatens to undo many of the global gains achieved with difficulty over the past 50 years.

The National Tobacco Control Program (NTCP) was launched by Ministry of Health and Family Welfare, Government of India in 2007- 08 to bring about greater awareness about the harmful effects of tobacco use and to facilitate effective implementation of the tobacco Control Laws. The National Tobacco Control Cell (NTCC) is responsible for overall policy formulation, planning, monitoring and evaluation of the various activities. National level public awareness/mass media campaigns for behavioral change are planned to be carried out.⁶ A number of countries have legislation restricting tobacco advertising, and regulating the sale and use tobacco products, and also, where people can smoke. One of the important legislates by Government of India, in 2003, to control tobacco use is the Cigarettes and Other tobacco products (Prohibition of Advertisement and Regulation of Trade and Commerce, Production, Supply and Distribution) Act. The Act is applicable to all products containing tobacco in any form i.e. cigarettes, cigars, bidis, gutkha, pan masala, khaini, snuff etc. A target of 15% relative reduction in current tobacco use by 2020 and 30% by 2030 has been set by National health policy-2016.⁶

The four major provisions of COTPA include:

Section 4: Prohibition of smoking in public places;

Section 5: Prohibition of direct and indirect advertisement of tobacco products;

Section 6(a): Prohibition on the sale of tobacco products to and by minors

Section 6(b): Prohibition on the sale of tobacco products within 100 yards of educational institutions;

Section 7: Display of pictorial health warnings on tobacco products

However, even a decade after enacting this law, its implementation remains sub optimal and variable across the Indian states. A national survey (2009-10) revealed that 29% of adults were exposed to the second-hand smoke at public places. About two in three adults had seen tobacco product advertisements. In 2009, 47% of youth were purchasing cigarettes from stores and 56.2% of them were not refused the sales despite their young age. However, 74.4% children reported seeing cigarette advertisements on billboards.⁷

Karnataka is one of the focus states under the National Tobacco Control Program. Karnataka enacted the state law (the Karnataka Prohibition of Smoking and Protection of Health of Non-smokers Act, 2001) even before COTPA was enacted by the national government and is among a few states that have shown political will by taking steps towards COTPA implementation. Since 2007, several districts in Karnataka have declared themselves as highly compliant to COTPA.

COTPA survey has been undertaken to assess the compliance to COTPA sections 4, 5, 6(a), 6(b) and 7 in Bengaluru urban district and Kodagu district with funding from NTCP.

App based collection system smart phone data was used to replace paper-based data collection for a COTPA survey in Bengaluru urban, and compared for the quality, cost and timeliness of data collection with the paper-based system in Kodagu.

To compare the data quality, timeliness, and operating costs of the smart phone data collection system versus the pen-and-paper data collection system in two districts of Karnataka.

Materials and Methodology

It’s a Record based comparative study. In Jan 2018 to March 2018, State Anti-Tobacco Cell together with the regional medical colleges conducted COTPA compliance assessment survey for Section 4, Section 5, Section 6a and Section 6b in Kodagu and Bengaluru urban districts of Karnataka.

Each district consists of number of administrative blocks. For the purpose of this study, each administrative block will be considered a cluster. The research team estimated that in each district, total number of public places vary from 6000 to 1 lakh Hence, the total sample size calculated at a confidence level of 95% on a compliance rate of 70%⁷ using OpenEpi software. Keeping in mind the density of population and public places in each district (rural and urban distribution), 2-3 clusters will be selected for the samples. Sample size are sub classified into 7 major type of public places, Point of Sales and Educational Institutes.

After getting the required number of sampling units (Public places/Educational institutions/Points of Sale), the field investigator will observe each of the selected unit by the transect walk method. Each field investigator will identify a fixed central point in each administrative block and follow a survey pathway; will walk south, east, north and west; and observe compliance to each section of COTPA and fill the applicable checklist as described in Table 7. This process will be continued until the recommended number is not obtained.

Kodagu districts sites were surveyed using a paper based semi structured questionnaire. Bengaluru urban district surveyed with same questionnaire using app for smart phones running on android OS.

The completely filled in paper-based records were identified (which had 351 records) from Kodagu district. Using computer generated random numbers 351 smartphone-based records were selected from Bengaluru urban for comparison as urban has 880 entries.

Data were coded and entered into Microsoft Excel and analysis was done using SPSS version 20.0.

Results

Paper based data collection

For every section, the investigator assigned unique identification number and completed the paper questionnaire. The filled questionnaires were then sent to Medical College on a daily basis. In the data office, data clerks entered the data manually into a Microsoft Access database that had data entry checks in place. A data analyst then went through the data and performed systematic quality checks by running scripts to flag errors and inconsistencies which were then reconciled by verification with the hard copy forms. Depending on the workload, questionnaires were at times batched before being entered into the database.

App based data collection

In this system, each section was assigned a unique identification number and data were collected through entry into a smart-phone using the Android app with touch screen features. Some of the programmed checks are put required to answer some questions in order to move forward in the survey were put in the app. In addition, range-value restrictions were established to prevent out-of-range entries for date of data entry. If an unacceptable response was entered, (for example, a response inconsistent with a previous entry) an error message would appear and the officer had to recheck the response and correct the inconsistency before continuing the survey. Once data was uploaded into the server, the data analyst performed quality assurance procedures to evaluate data completeness and duplication of unique identification numbers.

Completeness and logical consistency of data

We evaluated the completeness and the percentage of erroneous and inconsistent responses in the questionnaires. For pen-and-paper records, we used the original responses recorded by the investigator prior to data entry and cleaning. For each question, we identified and counted missing answers, inconsistent responses and also identified out of range values.

In the pen-and-paper data set, we separately determined the completeness of those questions that required a mandatory answer on the smart-phone application. We used Poisson regression analysis to compare incidence rate ratios (IRR) of the occurrence of missing answers and errors in the smart-phone and pen and-paper datasets, respectively. We adjusted the IRR for person and survey site location as potential confounders. A p-value ≤0.05 was considered significant in all the statistical analyses.

There was a higher incidence of incomplete records, errors and inconsistent answers in the pen-andpaper data (3.46%) than in the smartphone data (0.2%).Adjusted incidence rate ratio (aIRR): 7% CI: 4.4-10.3) with P value <0.01.

Timeliness of data

We compared median time from data collection to data entry into central database for the two systems. This was done by determining the date when data was collected and subtracting this from the date when data was entered (for pen-andpaper method) or uploaded into the data base (for smartphone method). Wilcoxon signed-rank tests were used to assess the difference between the two medians.

Smartphone-collected data were uploaded in less than 1 min (range 0.2 sec to 3.9 sec) into the final data collection sheet.

It took a median duration of 7 days (range 1–13 days) to have pen-and-paper records entered into the final data collection sheet (p<0.01).

Estimated costs of running the two systems of data collection

We estimated the cost of starting-up and operating each of the data collection systems based on costs established by the Technogatways company. Costs were categorized into start-up costs (if they were only incurred once, like the cost of application) and running costs (if they were recurrent, like the costs of paper and phone connection fees).

The estimated start-up cost for the smart-phone data collection system was higher (Rs 17,500) than that of pen-and-paper data collection system (Rs 15,500).

However, costs for the smart-phone system were less than that of pen-and-paper system (Rs 8,350 vs. Rs 15,500 respectively) in subsequent survey.

Discussion

In any survey, data collection systems should be sustainable, easy to use and able to provide timely data and consistently maintain data integrity. Our study demonstrated that a smart-phone data collection system using Application outperformed pen-and-paper systems across each of these domains for COTPA survey.

Data collected using smart-phones were more likely to be complete and had fewer inconsistencies and errors compared to pen-and-paper data. These findings may be attributed to the availability of programmed quality checks for select questions in App based program and may not be specific to a particular smart phone platform per se. A study from Fiji that evaluated public surveillance data collected electronically using hand-held personal digital assistants (PDAs) found similar results on data quality and completeness. In this Fiji study, data quality was measured using error rates (logical range errors/inconsistencies, skip errors, missing values, date or time field errors and incorrect data type) as collected using penand-paper versus PDA. Similar to this study, electronically collected data in the Fiji study were completer and more had fewer errors than paper collected data⁸ . The ability to include quality checks on a data entry in electronic data collection system makes it more versatile compared to pen-andpaper data collection system⁹ . In addition, errors were further minimized by directly uploading electronic data into the database without going through the process of secondary data entry, which can potentially introduce additional errors.^10,11

Studies have shown that electronically collected data take less time to become available for analysis compared to pen-and-paper collected data.⁹ However, in our study, the time taken to have electronically collected data available in the database could have further been shortened. Poor network coverage in certain areas within the facilities necessitated that data be saved in the smartphone’s memory and later uploaded into the server at convenient places where there was good network coverage. Occasional server communication breakdowns may have also increased the time taken for this data to be uploaded into the database. Despite these obstacles, our electronic data collection systems still reduced the time needed for routine data to be available for analysis by two weeks.

The cost of establishing and running the electronic data collection system was initially higher than that of paper-based systems. This was largely because of the higher cost of electronic equipment and operating software. Similar findings have been observed in a study conducted by Thriemer et al. where capital costs of setting up an electronic data collection system were higher than that of establishing paper-based system.¹² However, the overall costs of running the electronic data collection system were indeed lower and became more economical than paper systems in subsequent survey. This can be explained by the elimination of the need to have secondary data entry and intensive data cleaning as has been explained in other studies.¹² Since survey platforms are likely to be ongoing by definition, programs may consider using electronic data collection systems that are more sustainable.

Although not formally evaluated in our study, the investigators reported that use of smart-phones to collect data was faster, easier to follow and more convenient as they did not have to carry the weight of paper-based questionnaires. They also reported need for less space to store their data collection tools.

Conclusions

A smart phone data collection system outperformed pen-and-paper system across all domains for COTPA compliance survey. Data collected using smart phones were more likely to be complete and had fewer inconsistencies and errors compared to pen-and-paper data.

Electronically collected data take less time to become available for analysis compared to penand-paper collected data. The cost of establishing and running the electronic data collection system was initially higher than that of paper-based systems. But later survey it was less than that of Pen and Paper data collection.

Recommendations

Electronic data collection using smart phones can be effectively implemented in routine health survey in a tropical and developing setting.

In this setting a metamorphosed system provided the users with more timely, cost-effective, and higher quality data.

Limitations

Our study had several limitations. The two systems of data collection were compared using data collected separately from different place at different time periods. One may argue that the Investigators may have improved on their data collection skills over time and hence fewer errors/ inconsistent responses by the time data were collected using smart phones.

While electronic data collection systems may have data quality checks, it is possible that the investigators may have keyed in responses that are not consistent with those given in the questionnaire. This would compromise data quality and could not be measured in this case, but would also be more related to the quality of data collection personnel rather than data collection platform.

The costs of establishing and running the two data collection systems were based on rates provided byTechnogatways pvt.ltd. It is possible that App costs and other rates may vary from region to region thus affecting generalizability of our findings in this regard.

The App used in our study was also limited in its ability to accommodate complex programs required for larger research studies. The numbers of branching options needed when designing a logical flow of questions are limited. This makes it difficult to display select questions for specific sub-populations. Lastly, the sustainability of electronic surveillance systems may be in part dependent on current availability of software programs. Other smartphone application software will now need to be evaluated to determine if they are suitable to collect survey data.

Acknowledgement

We would take this opportunity to acknowledge our deep sense of gratitude to the dean and director of Bangalore Medical College & Research Institute, Bengaluru and Kodagu Institute of Medical Sciences for permitting us to take up the COTPA compliance survey in the Bangalore Urban district.We would also like to thank the Deputy Commissioner, Bangalore Urban and Kodagu wholeheartedly, for his support towards the survey. We sincerely thank the District Health & Family Welfare Officer and the Programme Officer, NTCP cell, Bangalore Urban district and Kodagu. It would not have been possible to conduct the survey with ease without the use of smartphones for collecting data. This was made feasible because of Mr. Vivek Athreya and Mr. Chethan Kumar from Technogatways pvt. ltd., who developed the survey application based on android OS. We thank all the authorities for helping us in the survey. Lastly, we would like to thank the survey team members of the survey.