Reproducing computational research requires more than having access to lines of code or compiled software. Reproducibility often requires running computations on an identical processor, or using a now defunct operating system. But computer hardware and software become obsolete surprisingly quickly, making the replication of old computational environments difficult or impossible. The advent of cloud computing and virtualization technology has opened a promising opportunity to address this problem. A researcher could preserve not only data and the computational algorithms used to analyze it, but the entire computational environment in which his research was conducted. Future researchers could then use virtualization to precisely replicate that environment, whatever hardware changes the future brings. The power of virtualization makes it not implausible to envision a library of virtual machines simulating every physical computer across the history of computing. This grant supports a project led by Carnegie Mellon computer scientist Mahadev Satyanarayanan and university librarian Gloriana St. Clair to build just such a library, called the "Open Virtual Machine Image Library", known as Olive. Funds will support the technical development of the OLIVE platform, initiatives to reduce the resources required to run archived virtual machines, and the development of a business plan and long-term sustainability strategy.

In a conventional journal, the mechanisms for feedback on published articles are limited to a letter to the editor or direct correspondence with the author. As an increasing quantity and diversity of scholarly products are disseminated on the web, one could imagine much more efficient and constructively visible commenting mechanisms. Initial experiments in so-called "post-publication review," however, have fallen flat. Comment boxes on online articles and other research materials overwhelmingly lie empty. Perhaps comment boxes are the wrong tool. Rather than asking a reader to comment on a full article, a much more granular approach might fare better, allowing readers to comment on a particular point, equation, or assumption in a published work. Funds from this grant support the development of hypothes.is, a particularly promising effort to build precisely such a granular web annotation system. This one-year grant to Planetwork NGO will support the design, testing, and launch of hypothes.is, bringing an innovative new pilot platform to fruition that has the potential to reshape how researchers communicate and interact with one another and with online scholarly resources.

In March of 2012, Christophe Pйrignon and Christophe Hurlin unveiled RunMyCode (runmycode.org), a pilot platform for linking research datasets and code with scholarly articles. The site links published papers with a RunMyCode "companion website" that provides a real-time environment where researchers can rerun the computations reported in the paper and reproduce the experimental findings reported. Initially launched with 40 econometrics and finance papers, the platform is an innovative attempt to use the Web to enhance the reproducibility and verifiability -and thus the reliability-of scientific research. Funds from this grant support a project by Columbia University's Victoria Stodden, Chief Science Officer of RunMyCode, to expand and enhance the platform. Over 16 months, Stodden will test the RunMyCode model in a number of additional fields, including computational mathematics, statistics, and biostatistics. Stodden will also pilot integration with existing scholarly platforms, enabling researchers to discover relevant RunMyCode companion websites when looking at online articles, code repositories, or data archives. Additional funds support the development of a comprehensive business plan and funding strategy for RunMyCode.

Though working paper repositories have become integral to a number of fields, including high-energy physics and economics, the impact of working paper circulation on the actual practice and production of research is relatively unexplored. For example, does circulation of working papers on digital platforms actually improve the quality of the work, whether in revised drafts or in final published form? How do researchers decide what to spend time reading, given the lack of a referee system? Can usage data from working paper repositories predict ultimate publication in refereed journals and citation counts of articles after formal publication? Funds from this grant support a research project by Erik Brynjolfsson of MIT and his Ph.D. student Heekyung Kim aim to explore these and other questions using the Social Science Research Network (SSRN) as a case study. In order to isolate the impact of the circulation of research in working form, they will draw on SSRN's logs of web server traffic, working paper citation data, and full-text analysis of individual papers to compare the usage and citation of papers posted in bulk by departments as they join SSRN (which are often already published elsewhere) with that of papers that have evolved as working papers on SSRN in advance of publication. SSRN will also perform a randomized experiment of different search algorithms on the live site in order to better understand user discovery and filtering behavior. In addition to this research, grant funds will support a workshop to bring together publishers and platform developers with economists and other social scientists studying scholarly communication to discuss existing research findings and potential future collaborations in this area.

We know from server log analysis that a substantial and growing percentage of the readers of any online academic article arrive not because they are browsing a given journal or author, but through the results of a search query using a search engine like Google or Bing or Proquest. We know little, however, about how researchers decide which items in search results are worth reading or citing or about how these changing information discovery and consumption patterns influence the choice of where one publishes one's work. This grant supports work by David Nicholas and Carol Tenopir of the University of Tennessee to better understand the behavior of academics as both producers and consumers of scholarly literature, in particular the role that judgments of trust and quality play in choices of publication channel, citation, and time investment in reading new material. Nicholas and Tenopir have built a unique corpus of web usage data from a number of major publishers' online platforms, which they will mine for insights into user behavior. Patterns of behavior in that usage data will inform the design of a series of focus groups and a broad survey to investigate reading and dissemination channel choices, and a series of "critical incident reports" will drill deeply into the underlying motivations for citation by asking select authors to walk through the discovery of and rationale for each citation in their most recent paper's bibliography.

According to a 2011 survey by Philip Glandon, only 35 percent of the 20 most cited journals in the field of economics have policies requiring as a condition of publication that authors make the data they use in their papers available to others. This is worrying, since empirical research requires quality control and lots of checking. Without access to the primary data a researcher works with, the larger economic community is unable to replicate her results, evaluate her faithfulness to her methodology, or re-use her data for other projects. What's worse, compliance is spotty even at those journals that do require authors post their research data, with fewer than half of all authors publishing the required datafiles. And when authors do make their data available, the files they post are often useless, since there are no discipline-wide standards governing what should be posted, what metadata should be included, or how programming code, procedural records, or explanations should appear. Funds from this grant support a project by Peter King to develop a software platform that has the potential to ameliorate some of these difficulties. King has developed the DataVerse Network, a platform specifically for publishing, sharing, referencing and analyzing social science datasets. With Sloan support, King will create a pilot platform that will allow participating journal editors to use the DataVerse Network in their article evaluation process, giving authors a uniform, standards-based capacity to upload and store research data, which can then be used both by editors and reviewers as an article moves through the publication process, and which will subsequently be available to the wider scientific community post-publication. The project represents a promising avenue in which information technology may help transform for the better scholarly communication.

In early 2011, computational biologist Phil Bourne hosted a meeting at University of California, San Diego (UCSD) titled "Beyond the PDF," which brought together the emerging community of researchers, librarians, publishers, and developers who are rethinking scholarly communication in the sciences. The primary focus of the agenda was a discussion of the future shape of scientific articles. Presentations ranged from models for data or software publication to so-called "executable" papers, in which results are not simply described but are actually computed on the fly in live, adjustable figures. The initial "Beyond the PDF" meeting was unusually productive, bringing together a group of stakeholders to think creatively about scientific communication, and forming a nascent community that has continued to develop through a series of international conversations throughout the year. Funds from this grant support a second "Beyond the PDF" workshop, to be held in the summer of 2011. Support includes funds for agenda development and planning, as well as monies to hire a full-year staff member to focus on providing services to the growing community of scientists and technologists focused on thinking seriously and imaginatively about the future of scholarly communication.