BGD. Aave <> Bored Ghosts Phase 2 Recap

Development
1

aave-bgd-phase2



TL;DR

Recap of the 6-months engagement for services of BGD Labs and the Aave DAO, starting on September 4th 2023 and coming to an end on March 4th 2024.





Context: Aave <> BGD Phase 2

On September 4th, the Aave community approved via governance an engagement for technical services with us, for a duration of 6 months and with a scope combining continuation on technical maintenance of infrastructure, together with different improvements on all Aave ecosystem contracts/tools.

Since then, this Aave governance forum has acted as BGD’s blog, following the approach of Phase 1 of showing our work to the community as transparently as possible.

Now, with our current engagement coming to an end, it is a good time to do a recap of what has been done, and what is pending to be delivered in the upcoming weeks.





What has been done?

Even with the scope being smaller and a natural continuation of Phase 1, we have executed/participated in the following items during this period.


1. Security


1.1. GHO vulnerability

(Done just before the start of Phase 2, but after our Phase 1 was already finished)

We participated in the resolution of a problem found on GHO, by supporting Avara on the initial remediation and follow-up actions.

Additionally, we introduced the FreezeSteward contracts for the EMERGENCY_ADMIN roles to be able to freeze assets on Aave v3.

During the last months, different exploits have been executed in DeFi protocols using a similar technique of “inflating” claims of underlying from a tokenisation on top.


1.2. Inflation attacks become a problem on DeFi

We researched this type of attacks in relation with Aave, introduced different protective measures procedure-wise, and did introduce into the pipeline of the upcoming 3.1 different protections on the protocol itself to protect against them.


1.3. Stable debt vulnerability incident

On November 4th, we received a bug report of a critical problem potentially affecting production, related with stable borrow mode.

We coordinated the initial protective measures, researched and found additional impact than reported and designed & executed an action plan to permanently stop the threat during the following ~2 weeks.


1.4. Seal Wargames

We got approached by the SEAL org team via SigmaPrime to perform a security incident simulation exercise on a decentralised protocol at scale like Aave.

During the day of the exercise, we led it and coordinated all simulated actions/reactions with other participants of the ecosystem like Avara or the Aave Guardian.


1.5. Organise and coordinate a security procedure engagement with Spearbit

Part of our scope is to understand when extra security procedures are needed, look for the appropriate party and coordinate them, supporting the researchers during the process.

We coordinated an engagement with Spearbit and the end of the year, focused on different part of the Aave v2/v3 protocol, where we thought extra review could be beneficial.


1.6. Participation as reviewers of the Immunefi bug bounty program

In the context of the Aave <> Immunefi program, we:

  • Proposed to governance and coordinated the activation of the program.
  • Reviewed close to 40 reports since the inception of the program.
  • Defined an strategy for more efficient payouts (Request for Bounty Payout) to security researchers, while keeping disclosures as responsible as possible.

1.7. Review of more than 100 on-chain governance proposals

Part of our responsibility as Aave contributors is to help onboarding other parties in different tasks of the ecosystem.

During our Phase I, and consequence of the long-lasting partnership, we identified that Certora could be a good candidate for review of on-chain governance proposal, this way having 2 separate parties (us and Certora) reviewing proposal in different point of their lifecycle.

During Phase 2, we supported Certora’s onboarding for review tasks. By the beginning of this month, Certora had fully taken over the on-chain reviews.

However, until Certora was fully prepared, we still did the majority of governance reviews by ourselves, amounting to approximately 130 on-chain proposals.


1.8. Aave Forest

Aave Forest has been a project on which we reduced priority to include others, at least from the initial conception.
We have been improving our Aave internal monitoring, exploring external tools and thinking on different ideas to move Phase 1 of Forest, but we think these require some extra work to be up to the standards of quality and decentralisation equilibrium.



2. Aave v3


2.1. Preparation of new networks activation proposals

During this period, Aave v2 has expanded to 4 new networks: Base (done just before the start of Phase 2, but after our Phase 1 was already finished), Gnosis, BNB Chain and Scroll.

The work on those items includes the initial preparation of all smart contracts of the core Aave v3, together will all different components we have been introducing for the last 2 years. Once that is done, we have also created for each one of the networks an activation proposal, allowing for the Aave governance to factually enable the new pool, in a fully decentralised way.

Additionally, we have done important work in the background on 2 pending networks (zkEVM & zkSync), with the first being ready for activation proposal once stability and liquidity improves, and the second requiring some extra work on the tooling side to be in “ready” state for Aave.


2.2. Network analysis

We have performed technical analysis for multiple candidate networks during this engagement, some of them already activated after the positive outcome, and some still pending:


2.3. Aave CAPO (Correlated Assets Price Adapter)

In order to improve the security on the Aave oracle’s side, we introduced the concept of CAPO, an extra smart contracts layer of top-of-price feeds of certain assets (initially LSTs and stablecoins) defining price ranges, starting with the upper side.

The development and security procedures are finished, and CAPO will be activated via governance proposal (if approved) imminently.


2.4. Aave 3.1

We developed a new minor-mayor version of Aave, to be applied via governance upgrade on all networks.

Aave 3.1 will be unveiled during the following days in its own post, but we can confirm that pending to finish some extra security procedures, it is ready for release, once/if the governance procedures passes.


2.5. Aave Killswitch

Initially proposed as a theoretical risk protection by Gauntlet, we led the development of the smart contracts of the Killswitch system, following expected requirements from the risk side.

Extensive information about Aave Killswitch can be found on the upcoming associated forum post, and its activation governance proposal is planned during the following weeks.



3. Aave Governance v3


3.1. Governance v3 contracts

During Phase 1, we developed Aave Governance v3 and a.DI, two totally new systems to revamp governance and cross-chain communication on Aave.

During Phase 2, finally both systems were activated in production, first via an interim period we called Governance v2.5 and finally on 25th December, with the complete migration of all Aave systems to v3, on proposal 416.

Since then, more than 30 proposals have been created, voted and executed on Aave Governance v3, with the system worked perfectly well, with no incident. Considering the scale of the migration, we think that the process has been a quite big success and leaves Aave in a really scalable position for the foreseeable future, governance-wise.

After the v3 activation, on the smart contracts side we have spent time mainly on research, with 1 important feature to be unveiled in a separate post really soon.


3.2. Governance v3 interface

Together with the smart contracts infrastructure of Governance v3, we created and released a user interface for anybody to access in a decentralised manner via IPFS, run locally or for convenience in the hosted version on https://vote.onaave.com/.

Generally, the community has shown appreciation for the interface, and since the release, we have been doing multiple technical and feature improvements. More specifically the following:

  • Features.
    • Added current voting power in the address information modal.
    • Added viem fallback provider logic.
    • Migration of Viem and Wagmi from v1 to v2.
    • Added BSC and Scroll chains support.
    • Reworked cache system.
  • Fixes:
    • Tracking for zero nonce GnosisSafe transactions.
    • Refactoring of the gas-less voting configuration.
    • Performance improvement of the ENS data fetching.
    • Refactoring of the delegation logic.
    • Refactoring of the list of voters logic.
    • Dark mode style improvement.
    • Seatbelt links for proposal details screen.


4. a.DI

a.DI (Aave Delivery Infrastructure), even if quite independent, was the complementary system being released with Governance v3, to allow Aave to expand in the safest manner to new networks, together with having fully decentralised control on all current ones. It was activated during the interim Aave v2.5 Governance proposal, and is active since then, taking care of all bridging needs for Aave smart contracts.

For the last 2-3 months since its release the focus/improvement have been the following:

  • Research different improvements to the consensus mechanism.
  • Different (ongoing) improvements on setup scripts and tests.
  • Improved pre-production testing flow.
  • Added support for Scroll network, with native bridge provider.
  • Added support for Wormhole as bridge provider, as preparation for Celo or other upcoming Aave networks.
  • Prepare updates of some a.DI adapters for new versions of the underlying bridges integrated (CCIP v1.2.0, Hyperlane v3, LayerZero v2).
  • Gas management improvements and better metadata on adapter.

Additionally, we have released a.DI Monitoring on https://adi.onaave.com, a dashboard-like user interface where the community can check both high and low level details of a.DI under the hood.
The codebase can be found on https://github.com/bgd-labs/adi-dashboard.



5. Aave Robot

On Aave Robot, the work has been focused on the following:

  • We added support BNB Chain and Base networks, using the new Aave Robot Operator smart contract.
  • We introduced a new iteration of the Robot Operator fully compatible with Chainlink Automation V2.1, applied for now on new networks (BNB Chain, Base).
  • For networks where Chainlink Automation was not available like Gnosis, we have enabled the robot via Gelato Automation.


6. Safety Module


6.1. New stkABPT and stkGHO

Already some time ago, the community approved to move the migrated the stkABPT underlying from AAVE/WETH (Balancer v1) to a AAVE/wstETH (Balance v2).

As the codebase of the SM instances had important legacy since SM v1, and considering the need of an extra instance for stkGHO, we did an important refactoring to be used on all new SM instances GitHub - bgd-labs/stake-token. This is currently used for stkGHO and the new stkABPT.

Additionally, we designed and execute a strategy (governance proposal) for a seamless migration from stkABPT v1 to the new stkABPT v2, for users to both migrate the underlying and stake on the new system as simply as possible. Since the Aave governance approved the migration on proposal 21, the majority of stkABPT has been migrated to the new version by users.



7. Aave v1/v2


7.1. Aave v1 further deprecation

After more than 1 year of Aave v1 officially deprecated, we proposed to governance proceeding with further off-boarding steps, and did the Phase 1 proposal of this process.

Phase 2 is currently being voted, while Phase 3 will not be part of the Phase 2 engagement.


7.2. Aave v2

On the Aave v2 side, the focus was mainly on the maintenance side, keeping into account its instances security-wise and, whenever operationally optional, trying to apply improvements planned for v3.



8. Support to other contributors and partners of Aave


8.1. Review of >100 governance proposals (before going on-chain)

During the last 6 months, approximately 130 proposals have been created on the Aave governance, between v2 and v3.

Not counting the ones created by ourselves, we have reviewed more than 100 proposals for different teams contributing to the community, before they went on-chain. This has helped reducing to almost zero the need of any proposal cancellation due to creation mistakes.


8.2. Forum participation on non-BGD projects

We have provided from time to time our insights on projects outside of our scope in this forum, but with implications on the ecosystem that we think were worth it to highlight. Amongst others, recently we participated in the Cross-chain GHO discussion and in the Oval Temp Check.


8.3. Misc feedback to parties willing to enter the Aave ecosystem

Quite frequently, external teams contact us to clarify different technical aspects of Aave, for example technical requirements for listings or for Aave expanding to new networks. To the best of our capacity and scope of work, we have always tried to give all necessary information to these projects, together with connecting with other Aave appropriate parties (e.g. service providers).


8.4. Support on Balancer security incident

End of last year, a critical vulnerability was discovered on some Balancer pools, and from the Aave side it was possible to mitigate the potential damage by coordinating protective actions in relation with Balancer boosted pools.

During the incident, being an historical close partner to the Aave DAO, we supported the Balancer DAO closely, removing any potential impact on close to $100m.



9. Tooling/UI/Misc


9.1. Introduction of maintenance proposals

On a big collection of governance-controlled systems like Aave is quite frequent that small technical maintenance tasks are required, like adding consistency between all instances across networks, or address legacy components.

In order to not overwhelm the community with a full governance flow on those minor tasks, we introduced the concept of “maintenance proposals”, and planned & executed 9 of them during this period.


9.2. Risk Stewards Phase 2

After a successful pilot on Phase 1 (CapsPlusRiskSteward), we proposed expanding the scope of the Risk Steward smart contracts to multiple other risk parameters of Aave, in a more generalised way. The goal: reducing governance overhead and optimising the flow of Aave risk service providers, while keeping a trust-minimised setup.

Even if still not released, the new RiskStewards codebase is close to completion and we expect proceeding with their activation governance proposal on the following weeks.


9.3. Aave Seatbelt

For Aave Governance v3 we developed a new version of Seatbelt which behaves analog to the previous seatbelt-for-ghosts. There are some improvements to highlight though:

  • Various quality-of-life changes (decoding all values on reserve config, highlighting known Aave addresses, better types & viem, etc.)
  • We worked with Tenderly to resolve some edge case issues with their state decoder resulting in better reports.
  • For chains without Tenderly support a less in-depth report is generated via Foundry, analog to our visibility tooling on aave-proposals.
  • Improved CI: the CI run time dropped from 30min to 1-4min resulting in faster reports. Also the CI now utilises Tenderly triggers to generate reports right after payload creation, where previously a cronjob was running every few hours. This is quite important to achieve fast visibility on proposal creation.

9.4. Aave proposals

The aave-proposals repository was the go-to place for most AIP creators on Aave Governance v2, containing different helpers to make the procedure straightforward. By integrating most of our tooling, we streamlined everything AIP creation related and reduced the entry-barrier by offering simplified methods for testing common problems.

The newly introduced aave-proposals-v3 is the same, just better :wink:; fully compatible and adapted for Aave Governance v3. The improvements were the following:

  • Proposals generator. In the old aave-proposals we already shipped a generator to simplify proposal creation. On v3, we iterated on that and now:

    • More than doubled the amount of supported proposal types (from 4 to 10).
    • Automatically generate tests dependent on the proposal type.
    • Support typed config files for the generator, as requested by some of the service providers for usage on their own infrastructure.

  • IPFS. Even if executable Solidity payloads are the core of any proposals, the associated description uploaded to IPFS is a quite important component too, and historically problematic on the review process. We did the following improvements:

    • Most governance proposals first create a snapshot before the on-chain vote. Not all AIPs specify this snapshot, and even if they do it’s hard to extract from the text, therefore we added snapshot as a property to the metadata of the IPFS description.
    • One issue we noticed with proposals being created after their code has been reviewed and merged to the aave-proposals repository, is that if they reference the repo in the AIP IPFS, it creates a adverse situation, in which the proposal is no longer relevant for us, but by removing it we potentially break links on the IPFS markdown.
      To tackle this situation we now automatically replace references to the main branch with an immutable git hash at the time of merge.

  • Payloads deployment/proposal creation. In the previous version of aave proposals, creators had to first deploy their payload and then insert it’s address in the proposal creation scripts. As was seen in some of the failed proposals in history, this still left room for human error (e.g. copying the wrong address).

    Therefore we upgraded the scripts to rely on CREATE2, to properly predict the address on the script side, so no copy pasting is needed. As the CREATE2 factory used within Foundry is not supported on all chains Aave is deployed on, we integrated and contributed to David Racero’s catapulta-verify to create a utility package to streamline verification of these contracts.
    Big props to David on his work on Catapulta and tooling around it!

    Additionally, the usage of CREATE2 makes verification of payloads by other parties rock solid, as now everybody can download the aave-proposals repository, and re-generate the payload addresses based on the local code, this way achieving 100% certainty that the proposal contains the expected logic, including dependencies.

  • Proposals preview. In the previous version of the tooling the script would show a preview link to the aave governance located at app.aave.com which only previewed the IPFS part of the proposal.

    To improve on that, we created a new preview screen at vote.onaave.com which shows not only its description, but the full data of a proposal, including payloads to be executed on each chain & Seatbelt reports for each execution. The goal of it is to give more certainty and visibility to proposal creators, allowing them to avoid unintended mistakes.


9.5. Aave address book

The aave-address-book repository has achieved broad acceptance in the ecosystem, and it is now used in nearly every AIP going live to assure address accuracy.
Over the past few months, we have made various quality-of-life improvements, on top of the maintenance work (supporting new pools, new governance, GHO/GSM, etc.):

  • We created a website with a searchable interface for the address book.
  • Generate an eMode enum, so people no longer have to lookup for eMode numeric ids.
  • All libraries are now generated for both Solidity and JS.
  • For the JS users we now also export the most important ABIs for easier use via viem or typechain.
  • We now also publish a tokenlist.json for DEXes & aggregators.
  • The addresses generator was migrated to viem for better typing and therefore easier entry into the codebase.

9.6. Aave CLI

The aave-cli is a command line package that can be used to simulate certain things related to the Aave ecosystem, like creating tenderly forks, predicting IPFS addresses or diffing Aave configuration snapshots. Over the past months we:

  • Adjusted the functionality from Governance v2 to Governance v3.
  • Added multiple Governance v3 specific features (simulation, generating proofs and even a command line UI for the governance).
  • Added functionality to generate a Seatbelt report for a local payload.
  • Added a new command to diff Foundry storage snapshot JSONs.
  • Started publishing the package as a standalone library.




Items to still be delivered

We believe that same as the community accepts flexibility on our scope, when reasonable, we also need to have flexibility on our side and finish the delivery of projects in advanced stage of development.

For that reason, and even if they will still take some extra weeks, we will be releasing/following until governance proposal stage the following projects:

  • Activation proposal of Aave 3.1, once security procedures finish and we completely unveil their contents.
  • Activation proposal for Aave Killswitch.
  • Activation proposal for RiskStewards Phase 2.
  • zkSync network analysis.
  • Activation proposal for Polygon zkEVM, with positive network analysis and pre-approved on Snapshot.




Learnings from Phase 2

In perspective, we think Aave <> BGD Phase 1 was a big step up and challenge for procedures of the DAO: from expansion to other networks, to designing a new governance and governance bridge system, going through multitude of items that hopefully improved the quality of the DAO.

On the other hand, Phase 2 had a different goal, more continuistic: maintaining the quality on the DAO, but trying to also adopt from our side a more project-specific role, by supporting other contributors as best as we can.


During this engagement, we noticed what we think are very important aspects of the DAO:

  • The Aave DAO is more and more akin to a professional decentralised town square. The model of service providers (now known in the community as SPs) combined with other types of contributors (independent, delegates, professional third-parties engaged in one-off fashion, grantees, white-hats helping securing the protocol) is generally working pretty well.
    Yes, it has its inefficiencies, but it is important to remember that decentralisation is a very important goal of the community, with some inefficiencies being a direct consequence of this.

    We think that for a DAO like Aave with important maturity, there is a future on iterating over this idea.

  • More contributors are required, but also better frameworks. Organizational decentralization is a continuous improvement process, but far from being a simple one. We firmly support the appearance of new professional contributors, but scopes should be always carefully evaluated by the community.
    For example, we underestimated the onboarding effort required by a very knowledgeable party like Certora into the intricacies of reviewing governance proposals, which created still some overhead on our side.

    We think the community should work on this direction of contribution frameworks, for example having existing contributors helping new ones.

  • Coordinating security efforts is resource-intensive, difficult to predict, and requires deep, specific expertise and critical decision-making. In a system with so much at stake like Aave, security incidents of any type (vulnerability reports, detection of problems of any kind affecting production, new “trends” on DeFi attack vectors, etc) are heavily critical in all senses. During this Phase 2, it is safe to say that security aspects have determined importantly the “pace” of everything we did. Sometimes directly by having the entire team allocated on the management & resolution of some incident (e.g. stable rate bug report), others indirectly, like for example re-shuffling features planned to be introduced in order to implement others.

  • Incentives are misaligned on expansions of Aave to new networks. From the first TEMP CHECK until the governance activation proposal of Aave on a new network there is a lot of technical work done in the background: analysis of the new environment (technical, assets, risk), handling custom aspects on some of the networks, coming up with good initial configurations, etc.
    Aave going to any network has value for Aave, but it also has an enormous value for the network itself, as one of the biggest DeFi platform sets roots there with its technology, its expertise, and all the rest of integrations coming directly or indirectly with Aave.
    Right now, the cost of expansion is in some cases totally asymmetric, with Aave bearing it almost fully. Considering the quality that Aave brings with itself, we hardly believe that the DAO should expect well-defined direct/indirect benefits in order to expand to any candidate network.





Next Aave <> Bored Ghosts

The Aave <> BGD Phase 2 officially ends on 4th March, but as mentioned before, some extra releases can be expected as a result of it even after that date. Our support to the DAO in security aspects will also continue during that period.

At the same time, we think the Bored Ghosts still have a place contributing in this community, and same as we did before, we will be proposing some new engagement points pretty soon.

We appreciate the trust the Aave community has put on us during this Phase 2, and we are open to feedback in this post.

16 Likes