Assessing Infrastructure, Output, and Performance of Local Journalism Ecosystems

For this analysis, the notion of the health of a local journalism ecosystem has been broken down into three connected conceptual layers – a) the journalistic infrastructure; b) journalistic output; and c) journalistic performance. Each of these conceptual layers, and the data gathering and analysis associated with them, are discussed in more detail below. The analysis of these three layers of ecosystem health has been applied to three New Jersey communities. These communities were selected in an effort to maximize the diversity of types of communities represented in this analysis, within the obvious confines of being limited to three communities. These communities are substantially different from one another in terms of their size, demographic composition, and geographic location within the state. These criteria were identified as critical to the dynamics of local news in New Jersey within previous studies (Hale, 2013; Starr, Weingart & Joselow, 2010). Generally, Newark is the largest, poorest, and most ethnically diverse of the three communities, while Morristown is the smallest, wealthiest, and least ethnically diverse. New Brunswick falls somewhat in between, but with a demographic profile that is closer to Newark’s than it is to Morristown’s and with a population size that is closer to Morristown’s than to Newark’s. Generally, we expect the health of local journalism ecosystems to be affected, to some extent, by the characteristics of the communities in which they operate, though we have not put forth any explicit hypotheses within the context of this exploratory analysis. More detailed profiles of each community can be found in Appendix A.

Assessing Journalistic Infrastructure

A key dimension of the health of any local journalism ecosystem is the extent to which a community is served by organizations and/or individuals producing local journalism. This study begins by looking at the journalistic infrastructure in each of the three communities. We operationalize the journalistic infrastructure in terms of: 1) the number of journalistic sources in a local community; and 2) the social media presence of these journalistic sources.

A starting point for this assessment involves counting the number of sources of journalism within a particular community. Such an activity has become more complicated than it once was. Obviously the increased volatility of this sphere, brought about by the rapid technological and economic changes discussed above, is a factor. Keeping pace with the profile of any local journalism ecosystem is much more challenging in this time, during which various journalistic initiatives are rapidly entering and exiting this space.

Given the inadequacy of available commercial or governmental data sources, any effort to create an inventory of the sources of local journalism serving a community is, to some extent, an ad hoc endeavor. For this analysis, we established a concrete, multi-stage data gathering protocol, in order to provide as much clarity and transparency about the process as possible. The process draws from – and to some extent combines – approaches employed in previous research. It involves consultation with the most authoritative relevant directories available, and supplements these consultations with a systematic search and discovery process that involves both online searching and engagement with members of the communities being studied.

The source categories are derived from previous research that has sought to provide comprehensive portraits of the media or journalism ecosystems in local communities (e.g., Durkin & Glaisyer, 2011; Durkin & Hadge, 2010; Gloria & Hadge, 2010; Morgan, 2011; Project for Excellence in Journalism, 2010;). The broad categories for concern here are: a) Television; b) Radio; c) Online; d) Print.

It is also important to emphasize that the focus of this research is on the local journalism ecosystem, which is defined in terms of the geographic boundaries of the three communities being studied. Thus, this analysis is focused on the journalism sources that reside within, and are oriented around serving, the three selected communities (e.g., Lin & Song, 2006). This approach excludes journalistic sources based and/or focused elsewhere, or more broadly, which are accessible within these communities. The focus here is explicitly on local journalism sources based in and serving these geographically defined communities. The search process for identifying relevant journalistic sources is detailed in Appendix B.

Through this process it is possible to create an inventory of the available, active sources of journalism in a community. The sources identified for each community are listed in Appendix C.[1] Given that communities differ in size and resources, it is obviously important to not employ the raw number of sources as the relevant metric for the health of a local journalism ecosystem – particularly if the metric is going to be employed for any type of comparative analyses across communities. Larger communities presumably can – and probably should – support larger, more robust journalism ecosystems. Thus, utilizing population data, we computed the number of outlets identified per 10,000 capita to produce a comparable measure of the prevalence of journalistic sources in a particular community. This approach draws from similar approaches employed in nutrition research that examines the availability of food sources in particular communities (see, e.g., Powell & Bao, 2008). Work in this vein also has been an important source of inspiration for “media deserts” research (e.g. Ferrier, 2013).

Of course, in the contemporary journalism ecosystem, social media play a vital role in facilitating interconnectedness and sharing of journalistic content (Pew Research Center, 2014). From this standpoint, an assessment of the social media presence for each journalistic source has been incorporated into the analytical framework as well. Facebook and Twitter have emerged as the most prominent news sources in social media (Pew Research Center, 2014). Thus, for this level of analysis, each journalistic source was evaluated in terms of whether it has a presence on each of these two platforms. It is important to emphasize that the primary unit of analysis for each part of this analysis is ultimately the community as a whole, rather than the individual outlet. So, in this case, aggregate measures were calculated for each community. For instance, a community with 15 journalism sources would have a maximum potential raw score of 30 (number of sources potentially on Twitter + number of sources potentially on Facebook). The total count would be divided by the maximum potential score to determine the proportional presence of the community’s journalistic sources on social media. This measure is intended as a basic indicator of the overall social media presence of a community’s journalistic sources (more detailed analyses of social media activity are incorporated into the Output and Performance layers as well – see below).

[1] This source list was generated in the Fall of 2014 and may not reflect new sources of journalism that have since emerged in each of these three communities.

Assessing Journalistic Output

The logical question that arises from the Infrastructure assessment described above is: how much journalistic output does the infrastructure generate? Thus, the Output Layer is focused on assessing the aggregate journalistic output within a selected community, within a specified period of time. The question here is one of quantity (the qualitative dimension is taken up in the Performance Layer), as it would seem that a reasonable indicator of the health of a local journalism ecosystem is the amount of journalism that is produced for the community.

For this analysis, a one-week sample of home pages [1] and social media accounts (Twitter and Facebook) [2] for each journalistic source was content analyzed to determine the overall volume of journalistic output available on these platforms.[3] A total of 1028 stories and 1651 social media posts were analyzed across the three communities. Again, controls (per 10,000 capita) were employed for these output measures to account for variations in the size of the communities, under the logic that larger communities should generate more newsworthy activity and also be served by more journalism outlets. For this output, measures of concentration were calculated, using the well-known Herfindahl-Hirschman Index (HHI), to determine the extent to which journalistic output is dispersed across available sources or highly concentrated within a select few.[4]

It is important to emphasize that the methodological approach employed for this section’s analysis—and the section that follows, relies on the journalistic content available online, regardless of the outlet’s “native” platform. Thus, the journalistic outputs of daily and weekly newspapers, magazines, radio stations, television stations, and local cable channels all are assessed via their online content offerings in the same way that the outputs of online news sources such as community journalism sites are assessed.

This approach runs counter to the common assertion that certain types of legacy media (e.g., local weekly print publications, ethnic media outlets) remain slow to utilize the Internet as a means of disseminating their content. We believe that we are at a point in the evolution of legacy media and their place within the broader media ecosystem that this generalization likely no longer holds true. The economic and strategic pressures and incentives to have an online presence, combined with the inherent economic imperative to distribute content production costs across as broad an audience base as possible (Hamilton, 2004), we believe mean that the content available online can serve as a reliable indicator of the relative journalistic output across individual outlets, regardless of their “native” platform. The key term here is indicator, as we are not seeking to produce a comprehensive inventory of journalistic output, only a set of indicators that are conceptually and methodologically robust and that can be employed in comparative analyses across communities or over time. It is worth noting that data gathered on the three selected communities revealed only one journalistic source in each community that did not have a corresponding online presence. Further, a preliminary analysis of the web sites for radio stations serving the three communities found that the quantity of journalism available on these sites varied in a way that reflected the stations’ journalistic orientations (i.e., news/talk radio stations’ web sites containing much more original journalistic output than music stations).

[1] Using website home pages as representative content builds on the tradition of sampling a newspaper’s front page, which is the most likely page to be seen by readers, and also represents the news outlet’s judgment as to the most important news to the community (e.g. Benson, 2013).

[2] Here, a limited “constructed week” sampling approach was employed. Specifically, seven days of the week (Monday through Sunday) were randomly selected for the month of January, 2015. The specific days selected for analysis were January 2^nd, 8^th, 11^th, 14^th, 17^th, 20^th, and 26^th. Inconsistencies in the archiving of older social media posts by the two major social media platforms prevented producing a constructed week from the entirety of a calendar year.[3] The sampled week of home pages was drawn from February 9^th through February 15^th of 2015. While it is generally preferable to utilize a “constructed week” sample in content analysis (in which individual days of the week are randomly sampled from across an entire year; see above), the combination of time sensitivity associated with this analysis and the unavailability of systematic archives of the relevant home pages over such a time period prevented such an analytical approach from being employed here. Future iterations or expansions of this research should certainly seek to employ a more rigorous approach to content sampling if time and resources permit.[4] The HHI involves summing the squared shares of each firm in a market to produce a measure of concentration. It is expressed as follows: . In the case of this analysis, shares of total journalism output within a community (whether in terms of news stories on the web or social media posts) are used in place of market shares.

Assessing Journalistic Performance

At the Performance Layer, the goal is to provide indicators of the extent to which the local journalism ecosystem is producing content that addresses the communities’ information needs. Thus, at this stage the content identified in the Output Layer is analyzed to determine the extent to which it is original, (as opposed to linked or aggregated from other sources) the extent to which it is about the local community, and the extent to which it serves communities’ critical information needs. We use these criteria as an admittedly rough indicator of the complex notion of the “quality” (Lacey & Rosenstiel, 2015) of the journalism being produced by these sources.

Given the centrality of the notion of critical information needs (CINs) to the ongoing discourse about the performance of local journalism (Knight Commission, 2009; Waldman, 2011), the approach employed here builds upon this concept, and the research it has inspired (Friedland, et al., 2012). Specifically, in an effort to provide a relatively simple and straightforward indicator of journalism ecosystem performance, the approach employed here involves content analyzing each story/post identified in the Output Layer to determine whether it fits into one or more of the critical information needs categories identified in Friedland, et al.’s (2012) comprehensive review of the literature that was prepared in order to provide guidance to the Federal Communications Commission for future empirical work. Friedland et al. (2012) provide eight categories of community critical information needs. These categories, and their associated definitions, can be found in Table 1. These categories provide a comprehensive and relatively straightforward schema for content analyzing local news stories/posts. It is important to note that while the extent to which the news/information had a local orientation was a part of Friedland et al.’s (2012) category definitions, for this analysis, we have employed a somewhat modified approach, in which the notion of critical information needs applies to broad content categories (e.g., education), regardless of their geographic orientation, as we sought to be able to separate the assessment of whether a story addresses a critical information need category from the assessment of whether the story had a local orientation (see below).

Toward this end, each story/post also was content analyzed in terms of whether it was about the local community as well as in terms of whether it was original. The emphasis here on original content is intended to separate aggregation, linking, sharing, retweeting, and re-publication activities, in an effort to determine the amount of original journalism output being provided to individual communities (e.g., Pew Research Center, 2010). The emphasis on locality is employed in order to facilitate analysis of the extent to which the output of local journalism sources is oriented around the local community. Both of these criteria are fundamental dimensions of the health of a local journalism ecosystem.

Content analysis of the news stories presented on home pages and social media posts was conducted by three trained coders. Pilot tests for both the web site and social media content analyses were conducted in order to identify data gathering challenges and difficulties interpreting or applying the coding categories. The coding sheets are included in Appendix D. Google Translate was used to facilitate coding of foreign language content (both Spanish and Portuguese language content were part of the analysis). More details about the coding process can be found in Appendix E.

As has been the case at each stage, concentration across journalistic sources was used to determine the extent to which journalistic output was emanating from many or few local sources. And, as with previous stages in the process, controls (per 10,000 capita) were employed to produce comparative metrics that account for differences in population sizes across communities. With these data it is possible to compute the proportion of stories/posts that address critical information needs, as well as to focus on stories/posts that are original, or that are about the local community (or various combinations of these categories).

Table 1: Inventory of Community Information Needs (adapted from Friedland, et al., 2012).

1. Emergencies and risks

Individuals, neighborhoods, and communities need access to emergency information on platforms that are universally accessible and in languages understood by the large majority of the local population, including information on dangerous weather; environmental and other biohazardous outbreaks; and public safety threats, including terrorism, amber alerts, and other threats to public order and safety. Further, all citizens need access to information on policing and public safety.

2. Health

All members of communities need access to information on health and healthcare, including information on family and public health in accessible languages and platforms; information on the availability, quality, and cost of health care for accessibility, lowering costs, and ensuring that markets function properly, including variations by neighborhood and city region; the availability of public health information, programs, and services, including wellness care and clinics and hospitals; timely information in accessible language on the spread of disease and vaccination; timely access to information about health campaigns and interventions.

3. Education

Communities need access to information on all aspects of the educational system, particularly during a period when education is a central matter for public debate, decision-making, and resource allocation, including: the quality and administration of school systems at a community-wide level; the quality of schools within specific neighborhoods and geographic regions; information about educational opportunities, including school performance assessments, enrichment, tutoring, afterschool care and programs; information about school alternatives, including charters; information about adult education, including language courses, job training, and GED programs, as well as opportunities for higher education.

4. Transportation Systems

All members need timely information about transportation across multiple accessible platforms, including: information about essential transportation services including mass transit at the neighborhood, city, and regional levels; traffic and road conditions, including those related to weather and closings; timely access to public debate on transportation at all layers of the community, including roads and mass transit.

5. Environment and Planning:

Communities need access to both short and long-term information on the environment, as well as planning issues that may affect the quality of lives in neighborhoods, cities, and metropolitan regions, including; the quality of local and regional water and air, timely alerts of hazards, and longer term issues of sustainability; the distribution of actual and potential environmental hazards by neighborhood, city region, and metropolitan area, including toxic hazards and brownfields; natural resource development issues that affect the health and quality of life and economic development of communities; information on access to environmental regions, including activity for restoration of watersheds and habitat, and opportunities for recreation.

6. Economic Development

Individuals, neighborhoods, and communities need access to a broad range of economic information, including: employment information and opportunities within the region; job training and retraining, apprenticeship, and other sources of reskilling and advancement; information on small business opportunities, including startup assistance and capital resources; information on major economic development initiatives affecting all community levels.

7. Civic Information

Communities need information about major civic institutions, nonprofit organizations, and associations, including their services, accessibility, and opportunities for participation in: libraries and community-based information services; cultural and arts information; recreational opportunities; nonprofit groups and associations; community-based social services and programs; and religious institutions and programs.

8. Political Life

In a federal democracy, citizens need information on local, regional, county, state, and federal candidates at all units of governance, including: information on elected and voluntary neighborhood councils; school boards; city council and alder elections; city regions; and county elections; timely information on public meetings and issues, including outcomes; information on where and how to register to vote, including requirements for identification and absentee ballots; information on state-level issues where they impact local policy formation and decisions.