Raw data needs preparation to be useful for research. In some cases, what is needed is cleanup and normalization; in others, tagging or categorization of dataset elements. Depending on the domain and kind of data, computers can do much of the necessary work, but some tasks, due to fuzziness or complexity in the data, are currently beyond the bounds of computation. Much data prep requires human eyes, human minds, human judgment, and human labor, a daunting demand when the size of many modern scientific datasets is measured in terabytes. The Zooniverse project, an international effort initially based at Oxford University and now housed primarily at the Adler Planetarium in Chicago, offers a straightforward solution to this problem: divide the work into very granular tasks, gather a large crowd of science enthusiasts, and let them loose on the data.  The result has been remarkably successful.  Over the past few years the project has launched more than 20 citizen science research projects across meteorology, ecology, astrophysics, history, zoology, pathology, and geology and has attracted more than 1.5 million registered and anonymous users, including a core group of 15,000 dedicated volunteers who contribute at least monthly. Funds from this grant support the next phase in Zooniverse’s evolution:  enabling faster growth to meet the explosive demand for Zooniverse projects.  The Zooniverse team plans to decentralize their governance model, expanding their online platform to allow community volunteers to take part in core management functions while still maintaining peer review and oversight by the research community to ensure that the project’s high scientific standards are met.  The result will be a more self-sustaining and scalable model, which the Adler Planetarium is committed to maintain as a national leader in using citizen science to serve its research and educational missions.

Github, the collaborative software development and versioning platform, has become so essential to the software development ecosystem that scientists have begun experimenting with using it for the collaborative versioning and sharing of datasets.  Though the potential value is immense, Github was designed to handle software code containing thousands of lines per file, not tabular datasets containing millions of entries.  Large datasets of the kind regularly used by scientists grinds the system to a halt.  Moreover, tabular data, unlike textual software code, might exist in any one of myriad data formats ranging from comma?separated to Excel to SQL.  Funds from this grant provide support for the development of a solution to this problem, a Git-esque platform called “Dat.”  Created by open source developer Max Ogden, Dat borrows heavily from Github’s syntax and mechanics, but is optimized for large-scale tabular data and has been programmed to be able to translate seamlessly between  the wide variety of formats commonly used to store data.   Grant funds will support the hiring of two developers: one focused on core development and one focused on providing interfaces useful to researchers and on ensuring the system’s interoperability with existing scientific data repositories. Additional funds will support outreach and partnership building with stakeholders in the scientific community.

Since 2011, the Mozilla Foundation, developer of the popular Firefox web browser, has hosted a series of “Software Carpentry” boot camps developed by computer scientist Greg Wilson to teach basic software engineering practices to researchers who in a professional capacity were writing code to manage data but had never received any formal software development training.  The project has been a success.  Interest in the boot camps has been robust both in the U.S. and Europe and Mozilla has expanded their effort into a larger project, called the Open Science Lab,  aimed at collaborating with the scientific community to develop open source tools and other resources to aid in scientific research and collaboration.  Funds from this grant provide continued operational support to Mozilla for this project.

One of the more surprising difficulties of working with big data—more than a few hundred gigabytes—is the sheer difficulty of moving it from place to place.  Though the price of cloud and local computing has dropped and the availability of bandwidth has increased, the standard protocols for transferring data over the Internet (http and ftp) simply start to break down at that scale.  Errors multiply, requiring laborious file integrity checking and repetitive restarting of transfer operations.  There is, as yet, no satisfactory solution to the simple yet thorny issue of moving meso? and larger scale data from one computer to another. Globus, a data management tool developed by a team at the University of Chicago’s Computation Institute, offers a promising solution to these problems, allowing the seamless transfer of large datasets with none of the drawbacks of existing methods.  The project is currently pivoting from support by grant funding to a sustainable nonprofit business model based on both individual and institutional subscriptions (and has already signed up six major universities as charter members).  However, it is facing a catch?22:  The team needs robust marketing and customer support capacity to build up a customer base, but without a customer base they will not have the funds to provide marketing and customer support.  Funds from this grant provide temporary bridge funding to the Globus platform, enabling the project to provide top quality service while it builds a customer base and moves towards independent sustainability.