A technical issue has been identified in the GHO integration with the Aave V3 GHO pool and as a consequence, GHO as an asset has been temporarily paused by the Aave Guardians.
The GHO that is minted is not affected and there is no risk to any funds, and all other markets are unaffected and are operating as normal.
The community is already working on a fix that will be proposed as an AIP shortly.
We have supported @AaveLabs during the process of submission of proposal 307 and confirm the payload properly does the following:
RISK_ADMIN to a FreezingSteward, a contract that will allow entities with EMERGENCY_ADMIN (Aave Guardian) to freeze/unfreeze, and only that.All additional details will be published in the following days, together with next steps.
AIP has been submitted to fix the technical issue with the GHO integration with the Aave Protocol, making the reactivation of the facilitator possible.
Additionally, a borrowing limit of 35 million GHO has been introduced to help manage risk exposure effectively. In this way, the borrow cap can be adjusted by the RiskSteward based on the existing framework for agile changes on caps, when necessary.
Summary
This post is to provide further details to the community on the recent GHO pausing event which occurred Aug-25-2023 1:37:35 CET. A technical issue was discovered on the GHOVariableDebtToken which led to a series of remediations. To safeguard against potential misuse, the Aave Guardian paused the GHO reserve. Community contributors Aave Companies, BGD Labs and Certora coordinated with governance proposals on a technical upgrade of the GHO VariableDebtToken. No funds were impacted or lost.
Technical issue investigation
According to the investigation, a user could repay their full debt and afterwards be left a âdustâ balance (i.e. in technical terms with a non-zero scaled balance of the VariableDebt token even though their borrow state was simultaneously set to false). In this scenario, if a user borrowed again, the borrow state for the user would not be updated to true. This was possibly due to the GHOVariableDebtToken accounting having an incorrect precision. To remedy this, an upgrade was required to prevent possible repeated borrowings of GHO up to the amount of the supplied collateral / LTV permits.
Timeline
| Date / Time | Details |
|---|---|
| August 24th at 16:37 CET | The Aave Companies receives a notification from a user concerning the https://app.aave.com interface (âInterfaceâ) explaining that they were facing an issue to repay a GHO borrowed position from the Interface. |
| August 24th at 17:45 CET | Aave Companies begins investigation into reported bug submission. |
| August 24th 21:30 CET | Aave Companies identifies an issue with the GHO VariableDebtToken contract and begins technical assessment and risk analysis. |
| August 24th 23.21 CET | Aave Companies contacts DAO service providers BGD Labs to confirm the assessment. |
| August 24th 23:26 CET | DAO Service provider BGD Labs validates the issue and Aave Guardian is contacted. |
| August 25 1:37:35 CET | Aave Guardian initiates first-protection remediation by pausing the GHO reserve. |
| August 25 3:53 CET | AIP 307 Proposal created to unpause GHO and instead, freeze the reserve. |
| August 25 10:54 CET | Security service provider Certora contacted by Aave Companies and BGD Labs and disclosed technical issue details. |
| August 25 - August 27 | Aave Companies in collaboration with BGD Labs and Certora continue with security procedures and develop a fix for GhoVariableDebtToken contract. |
| 27 Aug 2023, 22:03 CET | AIP 308 is created for resolution of the identified technical issue in the GHO integration with the Aave V3 Ethereum Pool. |
| August 29 23:46 CET | AIP 307 Executed and GHO reserve changed from paused to freeze. |
| September 1, 2023 CET | AIP 308 is Executed and the investigation is remediated. |
Consequences and current state
The pausing and subsequent freezing of the GHO reserve had no impact on users, and no position(s) created bad debt.
AIP 308 was successfully Executed on September 1, 2023, and the investigation has been remediated. GHO now has a borrow cap of 35 million and the GHOVariableDebtToken has been upgraded. Additionally, moving forward, the Aave Guardian has been granted the ability to freeze the GHO reserve.
We would like to thank the community, BGD Labs and Certora for their efforts to resolve this issue quickly. A bounty will be proposed to the DAO for the community member who reported the issue on the Interface.
Thankyou for the transparency!
For clarity, explained in a non technical way, what was the worst case outcome of this bug/exploit? In other words, how serious was it.
Secondly, genuine question, how come this wasnât caught in the review stage by BGD or auditors? Was it a particularly novel issue?
Iâd also love to know how the problem went undetected by BGD or auditors during the audit process
Following the approval by the community of proposal 307, and in order to keep technical consistency across all the protocol instances, we will be creating an on-chain proposal to activate the same FreezingSteward on all the non-Ethereum networks.
This will enable everywhere a mechanism for entities with EMERGENCY_ADMIN role to not only pause (highly invasive), but to freeze a reserve (non-invasive, but really effective lever protection-wise).
To clarify our involvement in projects like GHO, led by other teams:
We review design and implementation aspects of projects around Aave when community contributors ask us. But even if during those reviews in some cases bugs are discovered, we are not doing security reviews, as that is not our role.
In the case of GHO, we did a code review and shared feedback with @AaveLabs , but as commented, in terms of design, architecture, and implementation.
@AaveLabs can probably elaborate more on the nature of the issue, but one of the reasons why is not trivial to detect is its non-deterministic nature: depending on pseudo-random data (e.g. time), the problem appears or not.
We have created proposal 319, introducing the FreezingSteward contract in all non-Ethereum networks, to keep consistency across Aave v3 deployments.
Voting will start in ~24h, participate 
From the investigation, a user with a non-zero discount on GHO debt could enter into a unique state by performing a full repayment of their debt while leaving a small residual amount of dust, due to a non-deterministic rounding issue. In this state, they were able to withdraw GHO tokens up to the permitted loan-to-value (LTV) limit repeatedly without adversely affecting the userâs position, potentially allowing them to access GHO tokens up to the capacity of the Aave Facilitator.
Given the complexity and non-deterministic nature of achieving this specific state this issue was not detected during the various stages of development and testing for the project. These phases involved comprehensive analysis, rigorous testing, simulations, and security reviews conducted by the Aave Companies, as well as contributions from the broader community, including entities such as BGD Labs, Sigma Prime, Certora, among others.
We are actively collaborating with Certora to enhance the properties and invariants aspect of the Formal Verification for GHO. This effort aims to better identify edge cases where factors from the Aave Pool and GHO interact in unexpected ways. The forthcoming Formal Verification contest for GHO contracts will play a pivotal role in driving these improvements.
Furthermore, it is imperative to extend our appreciation for the collaboration and diligence exhibited by the individual who initially highlighted an issue on the frontend, which ultimately led to the discovery of the underlying problem. Their proactive involvement, although not a direct report, was instrumental in bringing this matter to our attention. Subsequently, with their assistance and the ongoing cooperation of BGD Labs and Certora, we were able to pinpoint and, effectively, resolve the root issue.
In recognition of their valuable contribution to the protocolâs security and stability, we recommend that the DAO consider awarding a bounty of 50,000 USDC to this individual. It is worth noting that while their input may not align precisely with the conventional definition of a vulnerability report, their insights were of paramount importance in instigating an internal investigation.
When will you reopen the GHO minting since the title says âtemporarily-pausingâ but we 5 months in without clean pathway.
Please provide best ETA?