Auto Pull Request Review Bot
PR Review Bot is a friendly bot that automatically approves pull requests based on a user-defined configuration.I've created this tool during my time at Omio to save us some time.
The story
When I first joined Omio, the team I was part of had 7 developers including an engineering manager. A year later, however, a reorganization happened that reduced the headcount to 4 developers. As if that wasn't enough, our team also took over the responsibilities of another team that consisted of 4 developers. So we were down on numbers even though we now had more work to do. We had to maintain our current products and somehow continue developing features.
These products were single-page applications on our website that acted as both a product and a platform that the other teams would build on. Other frontend teams developed their own component libraries and integrated their components into our platforms. There were at least 5 teams that would frequently make changes in our codebases. Since we owned these codebases, we were responsible of the code quality and the reliability of these products. We had to review every code change because we were the ones who was going to maintain that code. If an incident were to happen because of that code, we were the ones to wake up in the middle of the night to fix it. Therefore, it was important for us to review everything. It was fine when we were 7 people but it got crazy when we became 4.
Maintaining all these products while also developing the features was becoming impossible. At least one of us was always busy reviewing PRs. We took turns every week and became the dedicated PR reviewer for that week. It was normal that that week's reviewer would not get anything done that week.
It was not sustainable
Eventually, we decided to brainstorm about what we can do to reduce this overload. My idea was to automate our PR reviews. I had seen some patterns in the PR requests we got so I thought it would be possible to auto-approve some portion of based on those patterns.
For example, some teams developed components that were fairly isolated. Those teams would update their component libraries and all we had to do on our codebases was to fetch the new version through NPM. Their PRs would only modify package.json and yarn.lock. Even in package.json, they would only bump up the version of the library.
Another type of PRs we received often was translation updates. These pull requests would modify some JSON files and change a version number in package.json.
We decided to start with these use cases and automate their approval process. It required the bot to check which files were changed, make sure the files are still intact JSON files, and enforce that only certain dependencies could be update in package.json. So I got to work and within a couple of weeks, I've created this bot. I thought the bot's use cases could be generalized. We could use the same bot in multiple codebases or the other teams could use it for whatever use cases they could imagine. Therefore, I made the bot configurable. It read a YAML file called .pr-bot.yml which included all these rules. You can view a simple example here.
Did it work?
The number of PRs we had to review dropped significantly. We started to keep up better. Eventually, more developers joined our team and we got back to developing features regularly. But we continued to use the bot. In fact, we integrated it everywhere. All the codebases we owned used this bot. The others found use cases and started using it as well.
9 months after we started using it, I decided to check the numbers to get a better idea. It had approved close to 1000 pull requests over that 9 months. That is 1000 distractions we avoided. Out of those 1000 approvals and their subsequent deployments, a grand total zero incidents occurred. I'd say it's successful.
If you think the bot might be useful for you or you're curious and want to check it out; you can check it out here https://github.com/omio-labs/pr-reviewer-bot.