Kathleen Gygi




| LaPlaza | News From the Street | Online Papers | Live Agenda | Speakers |
| Travel & Accom. | Administration | FastFacts | Other Things |


Uncovering Best Practices: A Framework for Assessing Outcomes in Community Computer Networking

Introduction
Does the community network really build a sense of community for the geographic entity where you are located? Is the Indian tribe stronger, are the local merchants online doing better business, is there more participation in the local government, do people feel better about the role of the schools or the jobs the library is doing because both are online? Or do you and your users use the online tools mainly as an onramp to reach the world and not face local problems and realities? (Cisler 1994, p. 36).

Most community computer networks (CCNs) have broader goals than simply providing network services. For example, many aim to promote local economic development or increase public participation in political decisionmaking. Despite a certain amount of rhetoric surrounding CCNs, it is important to recognize that they are not mere ly technological systems. They are an attempt, often at a grass roots level, to use technology to mobilize and enhance existing community resources and social and political networks.

CCNs are social experiments that are being driven by technological developments, fueled by a sense of urgency associated with the emerging information superhighway. Many communities are worried that without immediate action, they will be left without onramps to the superhighway. There is a widespread assumption among project developers that networking is universally appealing and useful, evidenced by a "build it and they will come" approach to project design. However, because CCNs are a fairly recent and evolving phenomenon, and most data on CCN implementation and outcomes is anecdotal, it is difficult to assess the real benefits of these projects or to determine what constitutes success.

This paper advocates for the incorporation of evaluation procedures in the planning and implementation of CCNs. Although there are many approaches to evaluation, this paper proposes a framework for assessing community outcomes. It focuses on two categories of outcomes- 1) community economic development and 2) political participation-that are commonly targeted by local networking projects. As the documentation on CCNs grow, there is an increasing amount of qualitative data available about how CCNs are benefiting individuals and communities. However, there is very little quantitative data being generated.

Recognizing the need for both types of assessment, this paper proposes measurable indicators that can be used to assess progress toward achieving project goals. As Miller (1996) points out, to make measurable is to make visible. Reported outcomes from CCNs are discussed are used to illustrate this approach although, as noted above, very little data is currently available. The paper also raises issues that must be addressed in any kind of analysis of community economic development and political participation outcomes.

Framework for Assessing Community Outcomes
The first month of operation of the La Plaza Telecommunity has been enormously successful....Since 6 December, when the switch was flipped, over 750 people have become registered users. Taosenos are signing up, on average, at a rate of 10-20 per day. The first sign of real success has been heard-the busy signal. (La Plaza online posting 1995).

Approach
During the Fall of 1994 and Spring of 1995 I conducted research on CCNs in order to identify common goals, objectives, activities, and expected outcomes with an eye to developing an appropriate evaluation framework (Gygi 1995). I collected project data that was readily available online or by request from project developers and conducted a literature review on research and evaluation of interactive telecommunications projects. I found that evaluation, if done at all, has been limited and ad hoc. What little formal evaluation has been done tends to focus on user satisfaction and impacts to network providers rather than on communities.

Project success is generally indicated through increasing usage statistics. As Beamish (1995) suggests, the most important short-term goals for CCNs are to survive and grow. Usage can be an important indicator of success in achieving these goals. However, there are a number questions about usage that are not being addressed. For example, are users representative of the community? What characteristics of the users are contributing to the success, or failure, of a project? Concentrating on usage is also problematic because some services, such as emergency services may never be used, or used only rarely, and still have tremendous benefits. Furthermore, usage says nothing about changes in behavior.

The Colorado Rural Telecommunications Project (CRTP), is a notable exception in reporting outcomes. CRTP developed an economic outcomes methodology for evaluating rural telecommunications projects. Project data has been collected for 13 communities over the past five years. CRTP has posted case studies and outcomes data, based on both quantitative and qualitative measures of success, on the World Wide Web. The site also includes an Evaluation Tool Set that can be used by other communities.

The framework developed in this paper is based on a strategic planning and development indicator approach used by community economic development practitioners. Strategic planning begins by identifying a range of possible futures, then narrows the focus to specific actions and dedicates resources to these actions (Bryson and Einsweiler 1988; Chynoweth 1994). In this approach, projects are viewed as sets of resources and activities.

The outcomes analysis framework does the following:

Tables 1 and 2 illustrate this approach. It identifies several activities that might be undertaken to achieve expected outcomes associated with community economic development and political participation goals. The table uses a Consumer Reports-type rating scheme combined with example indicators and outcome measures. For the sake of simplicity, the expected outcomes in the tables correspond to project goals rather than objectives. Goals are broad statements of values; they need to be linked to one or more objectives that state how the goals can be achieved. Whereas goals may not be measurable, objectives can be linked to quantifiable outcome measures. Objectives are targets with time frames attached. For example, a community economic development goal such as "Enhancing Local Employment" would be linked to objectives such as "Create five new telecommunications-related jobs within two years." Strategies for achieving those objectives are embodied in project activities, which in the case of CCNs are primarily online services.

The activities, outcomes, and indicators in the tables are based on my research on CCNs and readings on assessment of outcomes in community economic development and political participation. The indicators provide quantitative measures of progress towards a project's goals. Outcome measures can be used for both subjective and objective assessment. For example, measures could be created for the degree of community support for an online service as well as for number of community organizations represented online.

In choosing outcome measures for CCNs, it is helpful to ask if there is a model of networking benefits that links project goals to specific activities and outcomes. For example, what is the chain of events that leads from placing public access terminals in libraries, or providing dial-up access to municipal databases, to an increase in public participation in political decisionmaking? Or how do CCN activities contribute to an increase in networking skills that would allow community members to acquire skilled jobs? The following two sections briefly explore how CCNs might be expected to benefit communities in terms of community economic development or political participation outcomes, and discuss appropriate measures for the expected outcomes.

It is important to recognize that outcomes can be positive or negative. Although not discussed in this paper, an example of a negative outcome for community economic development would be an increased reliance in the community on outside resources and entities or the community's losing its local flavor and traditions by becoming too high-tech. The creation of new elites or restricted access to government information and services are examples of negative outcomes for political participation. If negative outcomes are anticipated in advance, they can often be eliminated or compensated for.

Community Economic Development Outcomes
Community economic development combines community capacity building and local quality of life improvement strategies with more traditional economic development strategies such as local business retention and expansion and job creation programs. Common goals include: creating quality jobs for current residents; achieving local economic stability; and building a diverse economic and employment base (Blakely 1989). Maintaining local control of institutions and ensuring that local residents benefit from development efforts are central to this approach.

CCNs potentially offer a number of benefits to community economic development efforts because computers and networking technologies can:

These benefits can be realized through new services and institutions offered online or by supporting offline activities. Currently, online services with direct community economic development benefits are primarily information services and informal interest-group networking. There is no reason, however, why business support services or job training and placement could not be implemented online. For example, "virtual" small business development centers or incubators could be piloted on CCNs. Telluride's InfoZone reports that local businesses are using the system as their own in-house email facility and conducting online conferences among their staff.

Currently, the majority of CCNs are nonprofit, volunteer efforts. Of the 23 projects that responded to a survey conducted by Columbia University students, 18 were nonprofits, two were sponsored by state government, two by city government, and one was for profit (Molz 1994). The survey found that nine of the 23 projects had no paid staff members, relying on volunteers; most of the others had less than three paid full-time staff members. Government-sponsored projects or those with heavy foundation support tended to have four or more full-time staff members. A trend toward professionalization was evident, with an increasing number of projects developing business plans and budgeting for paid staff.

Thus, zero to four jobs created seems to be a reasonable outcome to expect in the short-term at least. The CRTP projects report similar results. For example, the Telluride InfoZone has resulted in five part- time and two full-time jobs over a five year period. Other projects have created from zero to two part-time jobs. The Colorado Technology Institute, the major funder of these projects, reports that 40 full-time equivalent (FTE) jobs have been created over the past five years, at a cost of $9,334 per FTE. In addition, several of the CRTP projects have reported outcomes on new business formation and business relocation, but the numbers are small, usually involving one or two businesses. If these numbers grow, CCNs might want to look at indicators such as "% of businesses that are locally owned" or "% of business purchases made within the community."

Basic online services such as email and public forums may be seen as prerequisites for community economic development if they build capacity of community-based organizations, local businesses, and residents. Thus, to assess an outcome such as "Build local capacity" a project might want to look at measures such as "No. of community organizations that use email" or "No. of local businesses that participate in business support forums." Increasing numbers on these measures might be viewed as conditions for promoting development. However, no data on these outcomes were readily available.

Accessibility and representativeness are crucial issues for community economic development. Accessibility includes physical access, cost, and literacy considerations. Three measures relating to physical access might be:

The Columbia survey found a wide range in numbers of public terminals, from one at the public library in Blacksburg, Virginia, to 45 for the Heartland Free-Net in Peoria, Illinois (Molz 1994). There is a similar range in the number of dialup connections offered by CCNs. Santa Monica PEN's 20 public access terminals translate into approximately two terminals per 10,000 residents (1980 population3D97, 117; Guthrie and Dutton 1992). Twenty percentage of PEN's usage originates from public terminals (Schmitz et al. 1994). Population figures were not readily available for other communities. Costs to users vary, depending on how the systems are financed and their type of ownership. Most private BBSs charge subscription fees. Access to Free-Nets is generally free to users; Blacksburg charges a monthly usage fee; there is no cost to registered PEN users. Data on actual costs was not readily available.

Ideally, users will be demographically representative of the community. Since CCNs often target the entire community representativeness can be difficult to evaluate. PEN found that the demographics of its users mirrored those of Santa Monica residents, except PEN users tend to be more male and more educated (Schmitz et al. 1994). The Columbia survey found that network users come in demographic waves (Molz 1994). The first users are mainly white, middle-class, younger, personal computer/modem owners, and relatively well-educated. The next wave has similar demographic characteristics, but users tend to be older.

The exceptions to the pattern found by the Columbia survey were networks and computer centers targeted at traditionally underserved populations, e.g., members of poor Black communities, new immigrants, or the homeless and unemployed. In addition to overcoming access barriers, CCNs need to target outreach and services to a variety of potential user populations, particularly low-income and minority populations, if they want their users to be representative of the community. Measures of representativeness would include both the match between the demographics of users to the community's population and the presence of important community organizations online.

Political Participation
Active and widespread participation by citizens in political institutions and decisionmaking processes is a cornerstone of democracy. Whether this participation should be direct or representative varies depending on one's theory of what constitutes a democracy. However, for the purposes of this paper, I assume that participation applies potentially to all the important functions of government, including agenda setting, policy formulation, resource allocation, dispute resolution, exercising the police powers of the state, and preserving the public, health, safety and welfare. The indicators in Table 2 focus both on groups and individual citizens as important actors in political participation.

The use of networking technology in participation mechanisms offers similar benefits to those discussed above for community economic development. For example, CCNs might reduce the costs of participation, e.g., time, travel, personal risk, and improve the quality and quantity of participation, through improved access, better informed citizens, new forums for public debate and decisionmaking, etc. As was true for community economic development, accessibility and representatives would appear to be prerequisites for improvements in participatory mechanisms.

A CCN might target a specific participatory mechanism such as voting. For example, an online voter registration program might look at the "% of eligible voters who are registered to vote" as well as the "% of registered voters that vote." A broader political participation outcome such as "Decreasing distance between citizens and government " might be assessed using measures such as "No. of email messages sent to government officials" or "No. of government agencies accessible via the network."

For example, Santa Monica PEN users send approximately 200 messages a month to city officials; 40 city agencies are connected to the network (Schmitz et al. 1994). An additional measure might look at the amount of resources the local government commits to involving citizens and community organizations, for example, "No. of full-time equivalent staff members dedicated to outreach per 10,000 population" (see Berry, Portney, and Thomson 1993).

Many CCN developers believe that providing online access to government officials and information will result in a better informed citizenry and a more participatory society. There is some evidence for this. For example, PEN users were found to be more a ctive than the average city resident in seven types of local offline activity, ranging from attending City Council meetings to contacting city officials (Schmitz et al. 1993). Similarly, Arterton (1987) found that more citizens participated in participatory mechanisms offered via computer and telecommunications technologies, e.g., two-way cable TV and computer conferencing, than in conventional forums, but only incrementally so. However, we don't know whether these people were more active before they be came involved in technologically mediated participation.

Although some assume that an increase in participation of any type makes for a more democratic society, measures of quantity say nothing about the quality of participation. A measure of quality (or depth) of participation would indicate the influence of participation on political outcomes such as agenda setting and resource allocation. For example, a deeper measure of participation might look at the "No. of policy initiatives that originate from neighborhood organizations and community groups" or the "degree of concurrence between citizens and officials on policy issues" (see Berry, Portney, and Thomson1993).

Such measures of depth would distinguish between activities such as putting government information online that are designed primarily to make government more efficient, i.e., reduce the government's cost of doing business, as opposed to increasing citizen participation in determining the outcomes of governing. I would argue that the benefits of providing information are indirect rather than direct in terms of participation outcomes. However, there is very little data available about participation outcomes associated with CCNs.

Conclusion
Data requirements of the type discussed here change the content and structure of the discourse through which perceptions and understandings are formed and out of which formal decisions and actions emerge. They change the rules about who can participate, what are legitimate topics for discussion, and how the discussion must be conducted. (Innes 1990, p. 19).

Community outcomes analysis is appropriate to use in the planning and evaluation of CCNs because it shifts the focus from technology platforms to changes in the community, offering measurable social indicators as alternative criteria of success to performance calculations or usage statistics. Such a framework can be used to set priorities, identify opportunities, and think strategically about project planning and implementation, as well as to assess the effectiveness of specific activities and services offered by CCNs.

Even though most CCNs are low-budget, volunteer efforts there are tremendous resources in all communities that project developers can draw on to do assessment. For example, local planning and economic development organizations and community groups can provide information on community goals, objectives, strategies, as well as basic demographic and business data. They can help set agendas, build consensus and support for projects, and provide services and content for the network.

CCN developers must recognize that a there are a wide range of community economic development strategies and that a particular strategy does not work equally well for all communities. CCNs need to partner with community economic development groups and professionals. Furthermore, they must recognize that technology choices embody values and that these values will effect project outcomes. Project developers who are concerned with building democratic institutions will need to ensure that their decisionmaking processes are participatory and collaborative. Community-based organizations and other key stakeholders need to be involved in project planning and implementation. The degree of outside control and the representativeness of the organizational structure will likely influence the community economic development and political participation outcomes associated with the project.

As this paper demonstrates, there is not enough data available on CCN outcomes to say conclusively what constitute reasonable measures of success in terms of community outcomes. Furthermore, determining which indicators are appropriate for an evaluation of a particular CCN will require a process of negotiation and planning between project developers and the community. However, looking at what data is available may give some sense of the range of outcomes that might be expected. While waiting for the bo dy of data to grow, CCNs should strive for increases in accessibility and representativeness among their users. A systematic approach to evaluation would allow CCN practitioners to determine the extent to which community outcomes support the social chang e they envision. Furthermore, it would allow network developers to uncover "best practices" among the diverse and growing number of CCN projects.