By Zach Brand

So far in our discussion of the NPR redesign we covered our efforts on launch night as well as upgrades to the API. We now return to January / February of this year. The team started to formulate how we would go about accomplishing the stated goals of the redesign. Our first major process decision was to approach this project more Agile-like. Of course such decisions need to actually be understood and carefully implemented or any project team will feel they are in a cartoon:

Most of our previous projects had been very waterfall in nature with requirements being fully defined before coding started. While useful on many types of projects, these frequently had undesirable consequences that we wanted to avoid on this project: unnecessary code being written, requirements not fully understood, division between teams, compromised functionality and/or architecture, and goals at the end of the project differing from those at the end. Most importantly since we didn't really know what we were building, we wanted better flexibility to figure it out as we went. Agile was well suited to solve this.

As such we implemented aspects of scrum, but also adapted to what would work for us. We made sure we had an integrated team with necessary skills (Designers, Editorial, UX, App Dev) working together from day one. We did do some rapid prototypes, and requirements were more of a ongoing dialogue than firm written down specifications. We decided to divide the project into 30 day cycles. Certainly not sprints, but clear milestones that forced us to keep moving. This became a painful, but useful tool as it was a struggle not to put off 'decisions' when there were still months to go in the project. We used planning poker (there's got to be a 10 card in here somewhere), we had standing daily meetings. And we physically moved 20+ members of the redesign team so we could sit in one contiguous area.  Of these the last may have had the biggest positive effect on the dynamics of the project. The spontaneous conversations, and ease of communication likely had as much benefit as any other process change.

Given the duration of our cycles we had varied success with burn down charts. From a management perspective having effort per task turn out to be extremely helpful. When an accounting question came up regarding hours spent on different types of tasks -- the team did not need do do anything -- all the data was already there. That said, as we got into crunch time at the end of the project, and hundreds of items were being added and removed per day, this became an a extra-step for the team that we chose to largely skip. Standing morning meetings (10:02am!) were trying at times, but under the leadership of our project manager Constance Miller they became very effective at setting expectations, raising concerns, and quickly resolving issues. They also featured amazing camaraderie and a fair bit of mirth. We also made it very clear it was the team's job to make decisions and drive the meetings -- not management. We even took to using the effective, but unfortunate labels Chickens and Pigs. The Chicken's were very limited in their role in these meetings.

Ironically in our first cycle a large portion of what we did was not directly tied to our three goals, but was about changing the project management and Application development tools we had been using. We will discuss the specifics of the tools in greater detail in a future post, but we found that our existing task / project / bug / code repository tools were not up to our forthcoming needs. Among the changes included implementing Subversion and portions of the Atlassian tool set.

Jira became the guts of the project management. We may have under estimated the amount of effort required to fully set it up the way we ultimately wanted. Ironically we didn't know exactly what the effort was as we didn't have Jira to track it! While these tools have become core to our process and a massive improvement, the migration was difficult and effectively set the team back a month at the outset.

We also did not have a development environment that was flexible enough nor one that would scale to the additional developers that we brought onto the project. As such we overhauled our entire development stack and created a virtualized development environment using a six node Proxmox ( OpenVZ ) cluster running up to 40 virtualized environments supporting a variety of OS and application stacks. This led to slow performance in some initial configurations but after increasing RAM and better optimizing the number of virtual environments per physical server became a much used tool.

Once we had the tools and process in place, it came time to really drill down on our goals.

Coming Next:

Redesign efforts in mobile

By Daniel Jacobson, aka @daniel_jacobson on Twitter

As mentioned, our site redesign resulted in more changes than just the visual ones on the site. Another change that was part of this launch that we are very excited about is the significant updates to the API, as follows:

Major Additions

Transcript API
We are excited to introduce our new Transcript API. This API offers all of our transcripts dating back to May, 2005. As of today, we are opening up over 80,000 transcripts through this API, and this number will grow with every new radio story we produce. This API contains the same transcripts that are now offered on the new NPR.org.

DISCLAIMER: The transcripts that we do have are only for our main programs, including All Things Considered, Fresh Air, Morning Edition, Talk of the Nation, Tell Me More. Weekend Edition Saturday and Weekend Edition Sunday. The transcripts take a while to produce, so they are typically not available until several hours after the program is over. Finally, while we believe the transcripts are largely accurate, there are some cases where they may not align perfectly with the audio or have grammatical or spelling errors in them.

Added Even More MP3 Files
Today, our MP3 repository goes back to 2005, all of which is available through the Story API (to the extent that we had rights to distribute the MP3s). As of later this week, we will be back-filling our repository with MP3 files dating back to 2001 for Morning Edition and All Things Considered. The rest of the programs will be extended to go back to 2003. With this offering, we are providing about 200,000 unique MP3 files through the API totalling more than 15,000 hours of MP3 audio. And these totals will grow as we add MP3 files for new stories and as we continue to back-fill our MP3 repository over time.

Improved Query-Ability : Query Filtering
With this release, we have added the ability to apply a new parameter to your Story API queries, called "requiredAssets". This parameter can receive "text", "images", "audio" or any combination of them. This list will also expand over time. By using requiredAssets, your query will tell the API to only return those stories that have the specified asset. So, for example, if your query has requiredAssets=audio,images, only stories that have at least one audio file AND one image will be returned. RequiredAssets is not currently an option in the Query Generator but will be added soon. In the meantime, this parameter will have to be manually added to your query.

Other Enhancements or Changes

Topic Changes
For the new site, we modified our topic structure to better reflect the kinds of stories that we produce and the way our users will navigate the site. Some of our old topics were retired, others were renamed, while a few were split and moved. For all of these cases, as strong believers in maintaining backwards compatibility, we set up server-side redirects to point all killed topics to new topics that closely (or exactly) relate to them. No old topic ID should fail, neither in returning results nor in having those results be sensible for that topic. The Topic ID list (XML) in our API reflects all of our current topics. Retired topics will continue to be valid and available, although new stories will not be added to them.

You can see all of the changes to our topics here (PDF).

Thumbnail Images
Prior to this release, thumbnail images were only displayed in their own XML element, called < thumbnail >. That element still exists (for backward compatibility). Because of the new format for the website, we now offer a < medium > and < large > thumbnail as sub-elements, the former being a 75 pixels square while the latter is 90 pixels square. In addition to the changes to < thumbnail >, the thumbnails have also been included in the standard < image > output elements.

Story List Element
For NPRML queries, we provide a < list > parent element as a container for the stories that get returned. This container, in addition to the title and teaser information that it has always provided, now offers links back the original API call as well as to an HTML page (if a related one exists). These kinds of outputs help the API be more REST-ful.

Version Upgrade
All of these changes culminate in a version upgrade. The new version is .93.

In addition to the new features mentioned above, I also think it is interesting to point out that the new NPR site is heavily dependent on the API. The dependencies are both in infrastructure and in enabling more extensive ways for efficient and expedient content creation. These topics will be covered in other blogs posts in this series. In our next post, however, we will discuss how we got started on the technical implementation: new tools, new processes.

We are very excited about these additions to the API and would like your feedback. Do these changes inspire new mashup ideas for you? What else would you like to see offered in our API's?

(By the way, for more conversations about NPR's API and other technical advancements, follow us on Twitter at @NPRTechTeam.)

By Andy Carvin, aka @acarvin on Twitter

Yesterday's relaunch of NPR.org was the culmination of a 15-month odyssey involving dozens of NPR staff. We were so eager to roll out the new site, we decided to tweet about the experience. What started as a series of occasional updates, though, became a 12-hour marathon of more than 150 tweets that made it possible for the public to get involved in the rollout process.

Until recently, our use of Twitter to talk about behind-the-scenes tech activities was confined to various NPR Digital Media staff discussing their work on their personal accounts. As the months passed, though, we began to notice how many people on Twitter would inquire about new features or -- let's be honest - problems with the site. So earlier this summer, we set up a new Twitter account, @nprtechteam, to serve as a liaison with the public about the back-end of the Web site.

As we started work on the relaunch yesterday morning, Daniel Jacobson and I began to post tweets talking about various aspects of the rollout, from tech-heavy explanations about each phase of the process ("Date formats are incompatible. Need SET NLS_DATE_FORMAT before executing the script") to behind-the-scenes snapshots capturing the vibe at NPR HQ ("We now have an official NPR.org relaunch mascot. Amy Schriefer brought her pug to work today. She's going straight for the bagels."). Twitter users seemed to appreciate the play-by-play, as the account grew from around 400 users to over 1,000 over the course of the day. (And we're already at 1,200 subscribers today, too.) Much of the feedback we received focused on the transparency of the process, keeping users apprised of each step. This held true for NPR staff as well; given the relative chaos of the launch and the more than 50 people involved, the Twitter account served as a news feed for everyone internally trying to keep tabs on the rollout.

The most important aspect of tweeting the rollout is continuing right now -- getting immediate feedback on what's working and what's not. Unlike some of the other customer service mechanisms we have, Twitter can act like a real-time chat room, allowing us to have a back-and-forth with people to sort out specific problems and document them for the development team in case they require a bug report.

And let's not forget that Twitter can help us out when things aren't going as planned. Soon after our initial rollout at 10:30pm ET, our servers began to experience serious load strain, requiring that we pull back and roll out the new site across the Internet incrementally. Not surprisingly, we started to observe people on Twitter tweeting that they liked the site, while others replied to them saying they couldn't access it. We immediately began to reach out to Twitter users through a number of NPR Twitter accounts, letting them know what was going on and encouraging them to follow @NPRTechTeam for updates. What could have ended up being a lot of confusion within the Twitterverse mitigated itself rather well because we were open about the situation and had a conduit to explain how it would be resolved. Twitter users, meanwhile, retweeted our updates, passing along these status reporters to a wider community of people.

For those of you who didn't follow our Tweet stream in real time last night, you missed quite a ride. You can find a transcript of our tweets below, recorded for posterity. More than a few people commented on how it felt like watching NASA prep for a shuttle launch. While that may be a bit of an exaggeration, we appreciate the sentiment nonetheless. :-)

Continue reading "The NPR.org Relaunch, As Seen On Twitter" >

By Zach Brand

Just over a year ago we launched the NPR API. Last night we launched a redesign of the NPR.org Web site as well as substantive changes to the CMS and APIs behind it. Things are moving quickly at NPR.org!

You may have seen Scott Simon's video tour, and you may have seen the editorial note, but over the next couple of weeks we will be posting the technical story behind this redesign. In a series of posts to Inside NPR.org we will be discussing the changes we made to our processes, changes to our tools, and changes that are both visible and not so directly visible NPR.org users.

While our planning for a site redesign goes back for sometime, for a variety of reasons we did not formalize our goals and begin technical work until early in 2009. Our stated goals were:

1. Create better flexibility in how we could publish a Web site.
2. Update the design and usability of the site.
3. Improve editorial efficiency in publishing.

The first goal was to allow greater freedom in how editorial staff could present content. Previously, any major efforts to display anything differently required application development. The second was about aesthetics and usability, while the third was reducing repetitive tasks and improving tools for our editorial group. The team who was to implement these goals convened for the first time in early February. We set a date of mid-summer as a reasonable time frame to make significant strides on all three goals - even though we didn't really know what the end product was going to be. What we did know was that this was going to be the largest project for NPR Digital Media to-date. Beyond just the scope of the project itself, we were going to have to make changes to some of our tools and processes to succeed.

Six months of effort culminated last night. We kicked off preliminary migration efforts around 11am. We optimistically hoped to be done by early afternoon. Then the various realities of completely overhauling the Web site and our CMS hit. While there weren't any massive surprises, we had a bunch of time-intensive delays that resulted in large part of the team working for over 14 hours. Shortly before midnight, NPR.org had a new look, the API had new features and a bunch of internal tools had changed.

As we continue the series around the launch of the redesign, we'll be covering a variety of topics: changes to the API, changes to our development tools, changes to our publishing tools and more.

Next up: the NPR tech team tweets its way through the site migration, including our various trials, introspections, wit and wisdoms, and how our mascot Zelda kept us sane for last night's launch.

Today we launched the new NPR.org. To learn more about the changes, read the editor's note or watch the video tour.

We welcome your thoughts about the new site. Feel free to add your comments below.

Sincerely,

Kinsey Wilson, SVP & GM, NPR Digital Media
Dick Meyer, Executive Editor, NPR News

For a taste of what the new NPR.org is going to look like as of July 27, NPR's Scott Simon offers us a quick tour.

We are posting this unusual "editor's note" to give our NPR.org readers early notice that change is coming. When you visit on July 27, you'll see a brand new NPR.org. It will be brighter, lighter, faster, easier to use, easier to search and more fun to surf. We promise!

Now, we at NPR well know that our listeners and readers are different than other news "consumers." NPR people are loyal. You count on the consistency, quality and integrity of our news and programming every day and we appreciate that.

So we didn't change NPR.org lightly. We've listened to your suggestions - and a few complaints. And we've taken the first step to build a new NPR.org that is more, well, NPR-ish.

We want NPR.org to be your source for NPR news, analysis, arts & life stories and music that is always fresh and up-to-date, a source of unexpected delight and most important, a site that always upholds NPR's highest standards.

On the new site, it will be easier to combine listening and reading, to follow breaking news, to comment on our work and share it, and easier to find programming from your NPR station. Later this summer, we also plan to enhance our support for a variety of popular mobile platforms. Our goal is to make sure that anyone with a mobile device can enjoy NPR and our stations. We want NPR to be always on for you.

So please come back and check out the new site and let us know what you think.

Respectfully,

Kinsey Wilson, SVP & GM, NPR Digital Media
Dick Meyer, Executive Editor, NPR News

Later this month, the annual Open Source Convention (aka OSCON) will be hitting San Jose, CA. For this conference, NPR has teamed up with The New York Times to present Media Mashup, an event focused on exploring creative uses of content from the NPR API and Times APIs.

The day will start at 10:45am with a presentation by Derek Gottfrid on the Open APIs of The New York Times. My presentation on NPR, Open Content and APIs will follow Derek's in the same room, starting at 11:35am. Right after my session, we will go across the hall to hold our Media Mashup, starting at 12:30pm.

If you are interested in coming to the Media Mashup event, please let us know in advance by registering and giving us your thoughts on what you would like to discuss, review, code, etc.

For this event, we have provided several examples of mashup ideas in GitHub. They can be found on our Media Mashup splash page. Please feel free to explore these ideas, make them better, create branches, etc. And come to the session with ideas of your own as well. These mashups are really meant to just get the conversation/development started.

To keep on top of this event, ask questions about it, etc., please follow @NPRTechTeam and @TimesOpenon Twitter.

I should also point out that we will providing food and beverages for this event.
--Daniel Jacobson