Data cleansing & Transformation of Observational Scientific Data : A Case Study

Abstract

This paper investigates information quality as it pertains to observational scientific data. Specifically, we focus on presenting a case study for initial efforts related to data cleansing and data transformation of 20 years of behavioral, reproductive, demographic and ecological data on wild bottlenose dolphins located in Shark Bay, Australia. The Shark Bay dolphin population has been monitored annually by researchers since 1984 with over 13,400 surveys of dolphin groups, several thousand hours of focal follow data on individuals, and large stores of film data on both groups and individuals. It is the most comprehensive dolphin data set in research today. However, the data is inconsistent because of changing standards, variations in researcher methodology, missing data and data entry errors. To add to the difficulty, the data is scattered across multiple applications and data repositories. One of the goals of the researchers involved is to integrate the data into a single repository so it can be used for sophisticated data analysis and manual data merging can be eliminated from the data analysis procedure. After presenting our data modeling, cleansing and integration process in the context of the Shark Bay data set, we introduce a set of quality metrics specific to observational science data and used them to assess the information quality of the wild bottlenose dolphin data before and after the data cleaning and validation procedure.