Debugging bug bounty programmes

1. Perform a pre-review pass with machine learning tools

Prior to initiating a bug bounty programme, and especially when the project is already in development, organisations should take the time to review the code using machine learning tools to identify any quick-fix bugs.

Not only will this reduce the number of bug bounties to be paid, it will also allow the researchers to focus their time on searching for the more subtle and less obvious bugs.

“Machine-based tools are good way to speed everything up during a first pass, so the human has more time for the deep dives and to look at the more unusual things,” says Colin Tankard, managing director of Digital Pathways.

2. Enact strict versioning rules

One of the key routines a development team should adhere to is following a strict versioning system, thus ensuring the latest version is shared for review. Complications can arise if the development team is working on one version of the code while the review team is examining another.

As well as being inefficient, conflicting versions of code can cause confusion when researchers are referencing parts of the code that are potentially no longer there, or have been substantially changed.

Organisations need to ensure their development teams adhere to management-led, organisation-wide, versioning procedures. This needs to be performed in conjunction with confirming that the latest versions of the code are published for review, thereby ensuring that everyone is working on the same code.

“We have experienced where the code is being developed by an internal team and it only takes somebody to pull the wrong version or not to upload it in time, for it to start to become out of synch,” says Tankard.

3. Introduce bug bounties sooner rather than later

Ideally, like conventional internal testing, bug bounty programmes should be implemented as the code is being developed. This allows bugs to be spotted sooner, allowing them to be resolved with minimal impact on the overall coding.

Waiting until development is nearly completed carries with it the potential for deeply embedded flaws to be uncovered. This could require significant changes to the code, which could have been mitigated had they been discovered sooner.

4. Scale up the bounties

One of the greatest challenges when first introducing a bug bounty programme is that organisations can be inundated with results. This can be overwhelming and demoralising for a development team. To mitigate this, organisations should start with lower-priced bounties, thereby reducing the number of bugs likely to be reported in the early stages of the bug bounty programme.

As the bug bounty programme continues, and the more easily discovered bugs have been found, organisations can start to increase the bounties. This incentivises a greater number of researchers to spend more time examining the code for subtler bugs.

“We try to guide the bug bounty programme to areas that are particularly interesting or complex, by looking at how much we pay,” says Ludwig. “If a product is not getting as much attention as we would like, we increase the amount we will pay for specific issues.”

5. Gradually increase the number of researchers reviewing the code

Organisations can also consider opening the programme only to a limited number of researchers initially, thereby preventing a surge of reports relating to the more obvious bugs.

As with restricting the initial rewards, organisations can begin to increase the number of researchers examining the code for potential vulnerabilities as the bug bounty programme matures.

An extension of this would be to completely change the researchers examining the code once the number of bug reports start slowing down, allowing new people, with different backgrounds and skill-sets, to examine the code.

“The idea is to slowly ramp up not only the customer development but also the researchers,” says David Baker, chief security officer of Bugcrowd. “You train both sides of the market place to provide value to each other.”

6. Curate the results

Often the bugs that have been found will be duplicates of what others researchers discover at the same time. Rather than having the results fed directly to the development team, and potentially distracting them with the same bugs being repeatedly reported, organisations should ensure the feedback is curated.

It is also important to check that the submissions cannot lead to the introduction of vulnerabilities into the code. These could occur as a genuine mistake, but equally – especially in FinTech industries – potentially be deliberate malfeasance by malicious actors wishing to subvert the code.

“We score bug reports using the common vulnerability scoring system, which is an industry standard way of scoring those issues, and have a service level agreement process,” says Ludwig. “There is a rate at which issues must be resolved, based on the severity of the issue, we bracket them as low, medium, high or critical issues.”

7. Have the bug bounty programme form part of the development cycle

Having a bug bounty in place is one thing, but the feedback needs to disseminated in an orderly and concise fashion. Rather than bombarding development teams with endless notifications when bugs are found, distracting them and potentially causing another bug to be created, organisations should incorporate the feedback into team meetings to ensure the information is adequately disseminated.

“Two main challenges exist when integrating bug bounties into an existing cycle. First, a team doesn't know when new reports appear, so it's not possible to plan proactively,” says Tim Douglas, an operations engineer at DuckDuckGo. “Second, the reported vulnerabilities can be severe, becoming the top priority and delaying other work.”

Although organisations can distribute the feedback as a regular email detailing all of the discovered bugs, it may be better to introduce the feedback in a meeting, to promote discussion and encourage creative thinking.

The key point here is to ensure that the development team is regularly updated with the latest feedback in a concise format without causing unnecessary distraction. This will allow the development team to work more efficiently, by highlighting which parts of the code should be prioritised.

8. Ensure reported bugs are resolved swiftly

Not only can it be costly for organisations to pay for previously identified bugs that have not yet been fixed, but researchers can be left feeling frustrated and unappreciated when they are repeatedly finding the same bugs.

A strict development cycle ensures any reported bugs are swiftly resolved, without interrupting the developers with endless notifications. A code review meeting at the commencement of the week, discussing bugs that need to be resolved, allows development teams to plan when and how the bugs are resolved.

Also, scheduling time for debugging, or assigning part of the development team to focus on this, can further help with resolving bugs before the next version goes live.

“Even if you have been penetration-tested, if you do not have a means by which to take a vulnerability presented to the development team to have fixed and prioritised according to its criticality, any pen test or bug bounty is not going to be effective,” says Baker.

9. Maintain project confidentiality

One of the main benefits of relying on internal penetration testing is that it reduces the likelihood of exposure to malicious actors. However, with the appropriate security protocols, these risks can be mitigated.

Rather than utilising an open bug bounty platform, where anybody can view the code, organisations can use what is called a closed bug bounty platform, in which only certain researchers are invited to take part.

Researchers can be curated based on being positively identified, passing a background check, and/or signing a non-disclosure agreement.

While restricting entry to a bug bounty programme can limit one of their core benefits, namely a wide skill-set through the broad range of researchers, it does allow an organisation to have controlling measures in place to maintain project confidentiality and security.

10. Ongoing review, rather than short-term assessment

Ideally, bug bounty programmes should be an ongoing review throughout the lifecycle of the product, rather than a short-term review.

As many online applications have ongoing support and regular patches to update the code against the vulnerabilities, these cause the code to evolve and expand in ways that were unforeseen by the development team. These updates have the potential to create inadvertent vulnerabilities, which organisations may be unaware of if they do not have a bug bounty programme.

Furthermore, having a bug bounty programme as part of the update cycle allows for the code to be constantly reviewed by external parties. This, in turn, allows organisations to focus more of their resources on development projects, thereby allowing new products to be developed faster.

Having these new products available on the market sooner will allow organisations to cover the costs of their ongoing bug bounty programmes for their existing products.

Bug bounty programmes allow organisations to better manage their own resources and expand the potential talent pool of security researchers examining their code. However, these programmes need to be carefully managed and curated, so that development teams are not overwhelmed.

The key is for organisations to remain open to constructive criticism and accept that they will never – internally – spot every vulnerability. “One of the things companies hang up on is inviting random researchers to look at their security.  They want more control of the situation,” concludes Ludwig.

“When you have a bug bounty programme, you have some definition of what that engagement should look like. It actually ends up being more transparent, but also more controlled.”