Build Blog Posts

How We Moved from Subversion to Git / GitHub

Tim Ledlie — Tue, 10 Apr 2012 19:15:04 +0000

In February and March, 2012, Kiva moved from Subversion to Git. We're six years old and have used Subversion from the beginning. For a few years, there has been a growing urge to move to Git. Engineers had a mixture of reasons, but I think it was mostly wanting to use the newest, coolest thing. We didn't have a "Git wizard", but a few of us had used it before in various contexts.

GitHub is just a few blocks from Kiva in San Francisco, so I contacted them and we had a field trip. I had seen Scott Chacon speak at ZendCon 2011, and he seemed like a friendly, generous, and smart guy. Scott welcomed us into their office, showed us around, talked with us about GitHub and Git, shared our pizza, and answered our questions.

We finally decided to go for it. Our engineering leads designated me to own the project -- I had as much experience as anyone with Git. It took me two months to get us fully migrated.

Deciding on a Workflow

Our Subversion workflow was:

Code for about 7 days on the "main" branch
Copy that branch to a release branch, begin the QA phase
QA / regression / bug fixes for about 2 days
Do a production release, which meant to start pointing prod at the new release branch

Git enables a wide variety of workflows. We decided to keep our workflow as similar as possible while changing the revision control tool. As a model, we mostly used Vincent Driessen's "A successful Git branching model", with a sprinkle of GitHub Flow.

Choosing GitHub

For our workflow, we needed to designate a "central repository" equivalent to our self-hosted Subversion one. We decided to go with a GitHub Organization. This allowed us to have private repositories, manage our users, and access all of GitHub's awesome features including Pull Requests for code review / collaboration and its diff viewer (with comments). Also, GitHub has Issues for issue tracking, and we are considering using it instead of our current tracker, Redmine.

Migration Gameplan

At Kiva, we have a primary codebase where most engineers do their work. But we also have a number of smaller repos that only one or two people work on at a given time. We started by taking one of those repos and moving it completely over to Git, including:

Clone the repo from Subversion to Git
Put the repo on GitHub
Have the engineer spend the iteration working with the Git repo
Change all the deployment scripts to use the GitHub origin repo

In the end, we had a four-phase migration, each phase taking an iteration (two weeks):

Migrate one small repo / app, end to end
Migrate a second small repo, continuing to learn and iron out details
Prepare documentation, timeline, rollout strategy, and communication for migrating the primary Kiva repo
Primary repo migration which involved all engineers; (also migrate our other small remaining repos)

The timing worked out well since we did the main repo migration during Innovation Iteration.

Steps to Move a Repository

I followed these instructions to use git svn clone to replay the Subversion history into a Git repo and put it on GitHub.

For our main repo, I took the additional step of creating an svn post-commit hook to do agit svn rebase / git pushto keep the GitHub repo up-to-date while engineers continued committing to the Subversion repo. This allowed me to migrate over back-end pieces, like deployment scripts and CI, while engineers continued using Subversion.

Migrating the deployment scripts to use Git was fairly straightforward -- replacingsvn co / svn upreferences togit clone / git pull. The only gotcha I encountered was how Subversion allows empty directories but Git does not -- but there are well-documented workarounds.

Migrating a repository took place over one two-week iteration. In the middle of the iteration, normal work happened in Git, while production hotfixes happened in Subversion. When we created the next release branch in Git, I changed the QA deployment scripts to use it. Just before doing the production push, I changed the production deployment scripts to track the Git master branch.

Other Behind-the-Scenes Work

Our Subversioncommits triggered emails to engineers with files changed and diffs. I replicated this for Git using GitHub's Post-Receive URL Service Hook in conjunction with a small script I wrote to grab the related patch file from GitHub (see here for example) and email it out. When researching how to solve this problem, I was impressed by the options available for automating tasks in GitHub using the Post-Receive URLs, static patch file urls, and the GitHub API.

We use Redmine for bug tracking and have a syntax in our commit messages that triggers a homemade script to update Redmine tickets. Kevin, another engineer,replicated this using built-in Redmine change support and the redmine_github_hook. He also had to clone our repos onto our Redmine server and set up a cron job to keep them up to date.

One unexpected issue was related to Subversion revision numbers being sequentially increasing, whereas Git commit hashes are not. With Subversion, it's easy to look at a revision number, compare it to the revision number from when you deployed code to a server, and know if that code is on the server. I built a script to replicate this functionality, which essentially compares where two hashes occur in the git log.

We use a url prefix for static assets in our CDN. Before, this prefix had our Subversion revision number, so it needed to change to the Git hash, and add an entry for this in our .htaccess file.

Finally, I trolled our back-end systems for references to svn and proactively changed the important ones to use Git and have a Git repo. This wasn't too hard -- I talked it over with one of our ops guys, grep'd for "svn", and created a spreadsheet inventory.

Rolling Out to Engineers

Up until the fourth and final phase of the migration, my work had no impact on our 20 engineers except Kevin, who works on the first two small repos we migrated. I did three things to make the rollout as smooth as possible.

First, I created a wiki page with links to reading about Git, our workflow with diagram, one-time setup, commands to run during normal development, etc. Basically, the "Git at Kiva" bible.

Second, I created a sandbox repo on GitHub where engineers could experiment and learn in an environment virtually identical to the real thing.

Third, I identified one person from each of our engineering sub-teams to be a "Git Guru" -- a go-to person with Git questions. I chose people who were comfortable at least with Git basics. I had a meeting with them where I went through the wiki page and our workflow in detail and had them do some test commits and pushes to origin with the sandbox. We discussed a few important issues like "Are we going to encourage rebasing by default?" (yes, but be careful about "rebase hell") and "When are long-lived feature branches dangerous and why?" which helped work out some details and get buy-in from the Gurus.

From the engineer's perspective, the migration was a few emails showing them the wiki page, encouraging them to try out Git with the sandbox, telling them the cutover timeline, and then finally announcing that we were cut over. Some people got it right away, and some people had questions that the Gurus or I answered.

Next Steps

We've had some problems with the interplay between PhpStorm (the IDE that most engineers use) and our developer VMs:

Lots of files unexpectedly changing mode
Not being able to Pull using PhpStorm, as it complains there are uncommitted changes (when there don't seem to be)
Your working copy ending up with uncommitted changes that are from a recently pulled commit, but that you didn't do yourself

Once we cut our first QA branch, we had a number of engineers not understand the subtle differences between the Git workflow (do your work on the release branch, merge the release branch to development) and our old Subversion workflow (commit your changes to both the development branch and the release branch). People had similar problems understanding how to make production hotfixes. This has led me to periodically walk over to an engineer and quiz them / answer questions about our Git workflow and branches. This one-on-one time seems to be a good way to answer questions and get everyone on the same page.

Once we have settled in to using Git in a few months, I hope to write another post reflecting its impact on Kiva engineering.

Categories:

Updates

Tags:

git

source control

Kiva Engineering: Tools of the Agile Trade

Cailin Nelson — Thu, 03 Mar 2011 21:55:55 +0000

At Kiva, our development process is based on Scrum - with work broken into two week Sprints (we call them "Iterations"). We're good students of Scrum in some regards (we do release every two weeks, without fail), and bad students of Scrum in others (it seems like we still bite off more than we can chew in each iteration). Like most Scrum teams, the center of our universe is our ticket tracking system. We use Redmine - albeit a rather customized version. We've put quite a bit of time and effort into our ticket tracking system - so we thought we'd give you a tour.

Redmine Home Page - Dashboard

One of the biggest lessons I've learned as both a web developer, and an engineering manager is that everybody is fundamentally lazy. Users are lazy, and guess what... engineers are lazy too. If you want somebody to do something many times a day - you have to make it really easy to do. If you want somebody to truly internalize a message - you need to deliver it as often as possible. Hence, each engineer's Redmine home page is a customized one-stop shop. The left hand column shows a list of tasks - divided according to which code branch the task needs to be performed in. We typically have three branches going at a time - one out in real world, one in QA and one in active development. The right hand side shows three burndown charts. One for the entire engineering team, one for each of our sprint teams (we have four), and one for you, individually. The burndown charts show the burndown for the "most relevant" code branch at any particular phase in a Sprint. (Note - the burndown charts are done with High Charts. Try it, it's great. )

Click to see full size version

More Burndowns

For the burndown addicted, we also have a page where you can view the burndown charts for all active branches at once.

Click image for full size version

Point Distribution

To check out what each individual on a particular sprint team is up to, we have some bar charts that show a point distribution. The bars are stacked - purple for points that are assigned, grey for points that have been completed.

Click image for full size version

SVN Integration

One of most recent, and most effective tools is our SVN-Redmine integration. We built a simple SVN post-commit hook that takes information from our SVN commit messages, and then updates the relevant Redmine ticket accordingly. The Redmine ticket is updated with the SVN comment, as well as the list of files changed, and a link to view each diff using webSVN. This feature is particularly handy for reviewing production patches, making it trivial to make sure that everybody knows exactly what was changed to fix Bug #11214.

Click image for full size version

Lesson's Learned

In a feature-driven engineering environment, everything always feels like a rush. It can be hard to stop and take the time to build a feature for yourself, rather than your customers. (It took us a year of procrastination to build both burndown charts and our SVN integration.) With the advantage of hindsight though, I've never felt that it was time ill spent - in fact, my experience has been quite the opposite - the amount of productivity gained using a new "engineering tool" always seems to outweigh the cost of building it.

Categories:

Announcements

Kiva Engineering: Innovation Iteration

Cailin Nelson — Thu, 10 Feb 2011 21:55:55 +0000

At Kiva, we tend to attract and hire engineers with a strong creative, entrepreneurial streak. This aligns well with Kiva’s office culture - and with our overall mission - an innovative, technology-driven approach to alleviating poverty. However, as the team grows and we become more specialized in an effort to become more efficient, sometimes our day to day work doesn’t always give us an outlet for our creative sides. After a long project involving concepts like currency exchange loss, it’s easy to feel like you spend your entire working life chasing a few misbehaving pennies around the globe, and that your particular slice of the Kiva pie has become a little divorced from the glorious big picture. In June 2010, many of us were finishing up some pretty tough projects, and as a team, we were in a bit of a funk. We talked as a team about what we could do to restore our energy and excitement. Our solution was an idea called “Innovation Iteration”. Innovation Iteration was inspired Google’s 20 Percent Time - and we do indeed spend 20% of our time in Innovation Iterations. However, there are two important differences to the Kiva approach:

We all take our innovation time at the same time.
Our innovation time is an entire, two-week iteration - not an “every Friday afternoon” approach.

Essentially, an Innovation Iteration is just like any other iteration at Kiva - except that the engineers get to pick their project’s themselves. Also, we’ve developed a fun and effective way to help us choose the best project to spend our time on. This part of Innovation Iteration is loosely based on third-hand rumors we’ve heard about the prioritization process used at Pandora. It’s a two part process and it goes like this: Innovation Planning Phase 1

We all get together in a big room with lots of markers and lots of paper
We write our ideas for the iteration down - one idea per piece of paper
We assign each idea a cost, in points
The pieces of paper are taped to the wall.
Each engineer very briefly describes their ideas.
Then, we break for lunch.

Innovation Planning Phase 2

Each engineer is allocated 4 points with which they may “fund” any project on the wall.
The points are colored stickies,and you “fund” a project by putting your sticky on the paper.
The engineer commits to work on the project(s) that they fund.
No project may be partially funded. (E.g. a project valued at 8 points but with 4 stickies cannot be worked on.)

This last bullet point is what makes the process fun. When the stickies go up on the wall, we usually start out with a bunch of partially funded projects. That kicks off a round of negotiation and cajoling and bribery, that ends with a suite of fully funded projects.

Once the projects have been selected, we create tickets representing the work to be done in Redmine (our ticket tracking system), and then we get coding! After the coding and QA cycle, Innovation Iteration concludes with a session called “Demo Friday”. Demo Friday is a festive gathering with beer and snacks, during which each engineer briefly demonstrates what they have been working on in the past iteration. Each demo is between 1 and 5 minutes, and run the gamut from showing off actual code to dazzling displays of javascript wizardry. Demo Friday is a part of all iterations at Kiva, but the Innovation Demo Friday’s are particularly fun and impressive, as there is a notable increase in throughput during Innovation Iteration. Innovation Demo Friday’s are also the time at which we make the final go/no-go decision on a particular feature. We recently completed our third Innovation Iteration, and this seemed like a good time to reflect a little on how it’s been going. I asked the team for their thoughts - here’s a sampling:

How does Kiva benefit from Innovation Iteration?

It's keeps us nimble. A risk we face as Kiva grows is becoming too conservative to tackle tough problems. This is a chance to prove to ourselves we can still use creative, timely solutions to solve big issues. -- Jonathan Innovation iteration is our vehicle for delivering grass-roots features whereas the other iterations are used for delivering features developed from an organizational top-down approach. A balance is necessary, one cannot exist without the other. -- Sid

How do you benefit from Innovation Iteration? It's a refreshing change from the norm. Especially for those of us who plug away at large-scale projects over many weeks / months, it's nice to do something totally different for a change. You often end up working with someone you don't normally work with, which is great for sharing your expertise and just getting to know your co-workers better. +1 knowledge sharing and +1 morale. Not only do I benefit personally from learning what others know, but it helps me build better and more relevant stuff for my teammates. -- Tim What risks do you think Innovation Iteration poses? There's always a chance we bite off more than we can chew. This could mean we fail to even complete a project on time. Or, even worse, that the ongoing maintenance is more than we anticipated and drags the whole org into a mess that one engineer made. -- Jonathan Code that is not released but in the source tree can become stale and eventually bloat. -- Chris What’s your favorite part of Innovation Iteration? Getting together for the two planning meetings -- hearing everyone's wild and varied ideas, negotiating for what we fund. I haven't yet chosen to work on one of my own ideas -- I've always been more excited by other people's ideas. -- Tim

Our next Innovation Iteration is just around the corner in March. What do you think we should build next?

Categories:

Announcements

Kiva Engineering 2010 : A Year in Review

Cailin Nelson — Wed, 19 Jan 2011 21:55:55 +0000

2010 was a year of growth for Kiva Engineering. We started the year with 8 full time engineers (and 2 managers) and ended the year with 14 engineers - a growth rate of 75%. We also kicked off 2010 with a switch to a new ticket tracking system (Redmine), that allowed us to more accurately track exactly what we've been up to. Let's take a look shall we.... Kiva Engineering is an Agile shop, operating roughly on a traditional Scrum model with two week Sprints (we call them "iterations"). We use "tickets" to track both defects in the existing product, as well as tasks for new development. Tickets are created by customer service, other Kiva staff members, our product managers and ourselves. Here we show the raw number of tickets created versus resolved. My take away... there will always be more to do then we can possibly get to! Although demand for our services contains to outpace our capacity, we do continue to get more work done as time goes by. At Kiva, we track work in "points". Points are a unit-less measure of work that we use to measure how hard we think a task is (estimated points) and how hard it turned out to be (actual points). Like most Agile evangelists - we shy away from giving a rough conversion of points to hours. Engineers new to Agile find this strange and a little frustrating at first, but everybody seems to full into agreement within a month or two. This second chart shows the number of points of work completed per month. Overall, this has been on the rise - no surprise since we've also been growing the team! Which brings us to the third and final chart - the one which best captures our efficiency as a team. This chart shows the points completed per coder per month. This chart also tracks how the number of coders on the team has evolved. Although it is clearly expected that we complete more points as the team grows, it is less expected and pleasing that our points per coder is also gradually rising. This means that despite the continual on-boarding of new team members and the increased "noise" of more bodies in the office (more emails, more questions, more dogs...) we're also continuing to get more efficient. Time will tell if we can continue this upwards trajectory as we continue to grow. (Did I mention that we're hiring...?) Note - in the final chart December was calculated as a 3-week month - a minimum estimate of the time we each took off to spend with our families over the holidays.

Categories:

Announcements

Implementing a CSS Grid

Gabriel Castillo — Mon, 10 Jan 2011 21:55:55 +0000

The engineering team at Kiva is going through the process of examining the rich set of technologies available for improving user experiences on Kiva's website. One such technology is the CSS grid. I recently spent some time evaluating and reading up on some of the different options that are out there, ultimately whittling down to 4 CSS grids that seemed to represent a good spectrum of what grids have to offer.

After implementing mockups with 1Kb, The Golden Grid, YAML, and 960gs our team came to the following recommendations:

Do not expect to use any of the grids “as is” for a production site.
Pool the practices and styles of the different grids in order to create a grid that works to enhance design and not confine it.
Build up a grid in communication with the designer
Keep columns and classes down to a minimum by only creating what is necessary.
Maintain communication across the development team, building the supporting infrastructure before implementing visual changes.

Taking the Grids For a Spin The first grid I evaluated was 1kb, which weighs in at a very minimal 640 bytes, uncompressed. While I was inspired by 1kb’s brevity, I wasn’t crazy about it’s use of "row" and "column" divs. The Golden Grid was the second grid tested out. I like how the Golden Grid has separate grid, reset, and typography stylesheets. Combined, they come in at just around 4Kb and the grid stylesheet alone is just over 1Kb. Some minor things I didn’t like about this grid were its use of cramped 10px gutters and its use of a column’s width within the column’s class name. (i.e. class=”g640”). I liked that, unlike other grids tested, it did not rely on cascading rules when designating a column element ( e.g. I could do class=”g640” instead of class=”column g640”). Following The Golden Grid I tested YAML ( Yet Another Multicolumn Layout ) and 960gs. YAML has the most components of any of grids tested and while I don’t plan on using YAML, I do plan on going through YAML’s code again and using it as a learning tool. YAML uses a fluid layout and additionally takes advantage of media queries, both which serve as great tools for building a responsive layout. My final evaluation was of 960gs. This is probably the grid I’ve come across the most and heard the most praise about and after using it, I can see why. I felt that 960gs did a great job of offering robustness while at the same time avoiding many presumptions about the design. In the situations I have come across, however, I haven’t found much use for the push_xx and pull_xx classes, and Ideally, I would prefer not having to rely on a container class in my cascading rules. The following are some of the other CSS grids that were looked into:

YUI Grids - http://developer.yahoo.com/yui/grids/
Blueprint - http://www.blueprintcss.org/
Typogridphy - http://csswizardry.com/typogridphy/
Tripoli - http://devkick.com/lab/tripoli/
Baseline CSS - http://baselinecss.com/

Tags:

css