There is something new on NPR.org today.

Starting now, it will be easier for you to talk to us, for us to talk to you and for you all to talk to each other. We are making it possible for anyone who registers with us to comment on a story and to create a profile page where many interesting things can happen. We are providing a forum for infinite conversations on NPR.org. Our hopes are high. We hope the conversations will be smart and generous of spirit. We hope the adventure is exciting, fun, helpful and informative. This is important for the NPR community.

That last phrase -- "important for the NPR community" -- is not phony baloney corporate rhetoric, I promise.

The NPR community is a real thing; it is made up of the people who work here, the people who work at member stations, the people who listen to NPR on the radio, the people who use NPR.org and the people who support NPR. And many in that community think of ourselves as "NPR people." Few other American news organizations inspire such allegiance, have a real community and have "people." NPR does and it is vitally important to our health and growth to be able to talk to each other more and more openly.

NPR is late to this game, to be blunt. Many big news operations have had open comments and other "social media" functions for quite awhile. Some of you are grizzled veterans of Facebook, MySpace, YouTube and online news commenting; for some this will be new. NPR has been cautious because we want to do it right; we want the comments and the conversations to be useful, friendly and civil; we want NPR employees to participate and talk about their work. We needed the right tools and the right philosophy to come together. Now it has.

NPR is a non-profit. We are not launching the project to get more "hits" that will make more money. We are doing it because it is the respectful thing to do for the NPR community. We expect to get story ideas, tips, insights into the world we cover, tough criticism and even the occasional compliment. We want to share more of the news we gather and the stories we tell with you. And we want to do all this in the NPR style -- with both dignity and self-deprecating lightness.

We won't hit the social media ball out of the park on the first swing. But we encourage you to create a profile and let us know what you like and don't like. We apologize in advance for any bugs you encounter. Also be sure to take a look at some of the more specific rules of the road. And if you don't like to do this stuff in public, here's my e-mail: editorial.director@npr.org.

--Dick Meyer

Over the last year or so, NPR has done a number of projects related to online communities and social networks, from Facebook to Flickr to Twitter. We'll continue to push further into services like these in a variety of ways, but we're also getting ready to bring it all back home with the launch of our own set of social networking tools on NPR.org.

Beginning the first week of October, we'll start rolling out a number of new features on the Web site:

User profiles. Visitors to the NPR site will be able to create a profile page for themselves. A profile page will let you upload an avatar, post a short bio and share your interests. This will also allow other users with similar interests to find each other. For example, if you say you're a fan of a particular band, you'll be able to click a link and see all the other people on NPR.org who like them as well. We'll also show you recent stories from the site related to that topic. Users will also be able to "friend" each other and post comments on each other's profile wall. We're also going to encourage NPR journalists and other staff to create their own profiles, so you can interact with them as well.

Discussion threads for all stories. Currently, discussion threads take place only on blogs, but we'll now have the ability to incorporate them in news stories. Each time a comment is posted, you'll be able to see the person's avatar, access their profile page, recommend their comment or even report it if you think it breaks the site's discussion rules. You can also sort the comments for each story by newest, oldest or most recommended. Meanwhile, site editors will follow the discussions to see what's taking place, so we can feature interesting comments either in the story itself, or even on the NPR homepage.

Story recommendations. Along with recommending comments in a given story, users will be able to recommend the story itself, not unlike the way you digg a story on Digg.com. This will let you explore stories on the site based on how often they've been recommended by the community.

Not too long after we roll out these features, we're also planning to launch a set of community tools not unlike groups on Facebook. We'll be able to set up community pages for shows and other NPR activities where users can start conversations in a discussion forum, upload photos and video, post event listings and the like.

We're really excited about rolling out these new features on the Web site. It's the latest step we've taken to open up the ways we interact with the public. NPR community members have always been eager to engage each other - just go to Facebook and take a look at the number of groups that have been created by members of the public. And we're eager to reach out to the public as well, using the Internet to foster new relationships between our journalists and the public. By creating new ways for that interaction to take place, we hope it'll impact the quality and diversity of our journalism in a positive way.

-- Andy Carvin

As promised, I wanted to give some history about how we ended up creating the NPR API. The first major decision that we were faced with was whether or not we should open up our API. The decision was not whether or not to build it, as we'd already done that. Back in November, 2007, we built the foundation of the API to launch with NPR Music. This is basically an XML file repository (essentially in an extended NPRML format) that contains all data needed to build pages on NPR.org. In addition to the XML repository, it includes a PHP framework used to render the XML files to the appropriate presentation layer (these layers include NPR.org as well as RSS feeds, podcast feeds, mobile sites and other outputs that we serve). Here is a diagram of the architecture which includes all of the caching layers as well, some of which were incorporated with the actual release of the public API:

Click image to enlarge

There are several reasons for this architectural approach:

1. PERFORMANCE : Requests will first go through the Memcache and file cache layers, which will always be the most efficient. If the requested document is not in Memcache, we have PHP render the output using the XML files. If the XML file cannot be obtained, PHP will access the database for the data. If PHP hits the database, however, a version of the request will be stored back in Memcache to speed up the delivery of the next request. This ultimately takes strain off of the database, which is the most expensive operation in serving documents.

2. ABSTRACTION : Creating a separate layer between the various presentations and the actual database allows the presentation layers to be agnostic with respect to the data repository. Currently, our database is Oracle, but if want to move to MySQL, then the presentation layers don't really care because they are served primarily off of the XML repository (although the final fail-over to the database would require changes).

3. SIMPLIFICATION : The database itself is a complicated relational system. The schema is largely normalized for scalability and efficiency in our write operations. Building pages, as a result, requires expensive table joins across very tall tables. These queries, although tuned, add up when you consider how many queries there are throughout a story page, for example. Executing these queries once and storing the data in a flatter file system enables the pages to be built more efficiently (both because of the flatter model as well as not having to access the database).

4. SCALABILITY : Because of the rendering framework, we are able to easily add new transformation and presentation layers without having to write a lot of extra code or customized database queries. The rendering engine knows how to handle the XML files in a cohesive way because they are relatively flat, so the transformation layers really aren't that different from each other. The framework also allows for reuse of code in the presentation layers because most of the presentations are dealing with the same content and are displaying that content in similar ways. New presentations for NPR.org are the hardest because of all of the design nuances, but adding Atom and MediaRSS are pretty quick and painless. The difficult part is figuring out how to map our fields to those structures, not in the coding of it.

So, the system was largely in place almost a year ago, alleviating many of the technical hurdles in building an API. We knew that if we wanted to open the API up to the world we would still have some technical challenges left, including filtering engines, the registration engine, the query generator, etc. Before getting to those tasks, however, we needed to determine if the public API fits with the overall NPR strategy.

-- Daniel Jacobson

Over the coming weeks, my colleagues and I will blog about the various decisions that we made while developing the API. The posts will discuss the following topics:

* Output formats
* OpenID
* Query generator
* Caching layer and performance
* Number of requests per user per day
* Audio stream vs. Download
* Amount and type of content offered
* Terms of use
* Rights
* Metrics
* Station content
* The archive and the deep NPR archive

I am sure that during the course of this series other topics will be added, but these capture some of the more prominent issues that were discussed. As you can see, these topics involve technical issues as well as legal and business ones.

Before we can get to any of the above topics, though, we have to address the single most important decision that we made: Should we open up the API?. That will be the first post in this series.

The purpose of this series is to continue to be as transparent as we can be and to be an active, engaging part of the technical community. We hope that some of these decisions that we dealt with will help others successfully pursue creating APIs as well. We also hope that this blog will act as a forum to continue the discussion and will help us continue to better deliver useful tools and services.

I am looking forward to the discussion!
--Daniel Jacobson