Aave v2/v3 security incident 04/11/2023

Development
1

Summary

Today 4th November 2023, we received a report on the Aave bug bounty program about a high vulnerability affecting Aave v2, which afterwards was raised to a critical.
After some coordination with the Aave Guardian, protection measurements that completely stop the attack vector on the whole Aave protocol have been applied, and all Aave pools are perfectly safe.


The reported vulnerability

Currently, all Aave pools are protected with the measures taken, but given Aave v2/v3 is a protocol “forked” by multiple third parties, we don’t think it is responsible to give full details of the vulnerability yet.

However, to provide maximum tranquility and transparency to the community, we have compiled the following FAQ.

What pools were affected by the vulnerability?

Some assets of Aave v2 Ethereum and Aave v3 Optimism/Arbitrum/Avalanche/Polygon were potential targets of the attack vector.

What exactly is the vulnerability?

As commented before, we don’t think it is responsible to disclose the details surrounding the vulnerability, but we can say that by disabling stable rate mode borrowing, it is not exploitable.

Was the vulnerability exploited?

No, we simply received a bug report and protected against it.

Any funds at risk?

There are no funds at risk at the moment.

What are the next steps?

  1. We are creating a governance proposal to remove the current freezing protections, and apply a more specific one: disable the stable rate mode for all assets which have it.
  2. In parallel, we are designing a plan for the unpausing of the affected pools.


Once we think it is responsible, we will publish an extensive explainer of the vulnerability and how was the course of action from disclosure to fix.





Updates


→ Nov 8th


For transparency with the community, now that all the major planned governance proposals of protection remediation have been created, the estimated timeline for every item is the following.

IMPORTANT.

  • This assumes the community votes for YES on all the proposals.
  • Due to how governance proposals work, execution timing can vary slightly, but in the order of low hours/minutes.

Proposal 358 Disable Stable Borrows

  • Created: November 04,10:30 PM UTC
  • Estimated execution time: November 10, 15:18 UTC
  • The goal was?: the first line of protection, stopping the reported vulnerability
  • What unblocks?: in practice nothing; v2 Ethereum will keep being paused as the following proposals are required before unpausing (for security reasons).

Proposal 359 Multichain Stable Debt Token Upgrades

  • Created: November 06, 2023, 09:30 PM UTC
  • Estimated execution time: November 12th, 2023, 07:30-09:30 PM UTC
  • The goal was?: full protection for the vulnerability of all assets being upgraded
  • What does it unblock?: it will be possible to unpause all assets on v3 Polygon, v3 Avalanche, v3 Optimism and v3 Arbitrum. CRV on v3 Polygon can’t be unpaused.

Proposal 361 Liquidations Grace Sentinel Activation

  • Created: November 07, 2023, 05:20 PM UTC
  • Estimated execution time: November 13th, 2023, 03-20-05:20 PM UTC
  • The goal was?: activation of the Liquidations Grace Sentinel feature for Aave v2, which risk providers can recommend using to give a grace period for previously paused assets. Additionally, upgrading implementation of extra v2 Ethereum assets and CRV on v3 Polygon.
  • What does it unblock?: Full return to operations on all pending Aave instances and assets (v2 Ethereum and CRV on v3 Polygon). If the risk providers recommend adding a liquidations grace period for any asset, the unpause of v2 Ethereum will happen just after that grace period for that specific asset only.


→ Nov 10th


As an update for the community, proposal 358 has been executed early today, following the timeline. This doesn’t create any meaningful effect on users, as all affected assets are still paused.

The next event will be on Sunday November 12th, 2023, 07:30-09:30 PM UTC, when assets on v3 Optimism/Arbitrum/Optimism/Polygon will be unpaused.

P.S. To keep clarity on this highly populated thread, we will move all BGD updates to the 1 post, with the time of publication.



→ Nov 11th


An update for the community, proposal 359 has entered into the last timelock of 24 hours and will be ready for final execution on all networks tomorrow Sunday, 12th.

As previously described, this will mean the following:

  • All assets on Aave v3 Polygon, Arbitrum, Optimism, and Avalanche will be eligible for unpause by the Guardian.
  • The unpause by the Guardian will happen slightly later than the final proposal execution. We will try to support them to be as close as possible once the proposal has been executed on each network.
  • Aave v3 has no Liquidations Grace Sentinel, so immediately after unpause, all operations will re-start. We recommend users closely monitor their positions if they need to take any actions on the unpaused assets.
  • Aave v2 Ethereum and CRV on Aave v3 Polygon will remain paused for 1 day more.

The estimated execution times for each payload (automated by Aave Robot) are the following:

Polygon
14:25 UTC

Optimism
14:04 UTC

Arbitrum
14:17 UTC

Avalanche
14:08 UTC

Unpausing can happen anytime after execution, whenever the Guardian can process the transaction.



→ Nov 12th


All Aave v3 instances (Arbitrum, Avalanche, Optimism, Polygon) have been unpaused by the Aave Guardian, after the successful execution of proposal 359.

Depending on operational aspects (recommendations of grace periods, and Guardian coordination), v2 Ethereum is estimated for tomorrow morning UTC, earlier than initially communicated due to block times on voting.



→ Nov 13th


As an update to the community, the liquidations grace period on Aave v2 Ethereum has been set by Guardian to 14:30 UTC , slightly more than the 3 hours recommended by risk providers from now.

Unpause of v2 Ethereum will follow, which will mean that v2 Ethereum will be fully operational again, but users with WETH will have a 3-hour grace period to protect their positions.



→ Nov 13th


Aave v2 Ethereum has been unpaused by the Aave Guardian. Again, if you are a WETH user, it is possible to protect your position.



→ Nov 16th


As the last step in the set of planned actions, we can confirm that Guardian has unpaused CRV on Aave v3 Polygon.

This means that all Aave v2 and v3 pools operate normally, without exception.

15 Likes
2

Thanks for the swift update @bgdlabs.

2 Likes
3

Following the plan, we have created proposal 358 on the Aave Governance which will:

  • Disable stable rate mode for all assets on the v2 and v3 instances where it is active.
  • Unfreeze all assets that were previously frozen by the Aave Guardian.

Voting will start in ~24, participate :ghost:
https://app.aave.com/governance/proposal/?proposalId=358

5 Likes
4

May I suggest an alternative because I will very much miss the stable rate?

For Aave, the stable rate is determined via a rough 2x calculation of the variable rate. Even Aave’s stable rate won’t stay the same in some outlier situations: Borrowing - FAQ . But if the lender stipulates upfront how much they are willing to accept no matter what the utilization rate is, then that works great for the borrower. And it is fantastic for shorting. To do it, Aave uses a behind-the-scenes orderbook.

Same interface as currently exists in Aave, just with extra functionality for this order book that happens after you click the “Supply” or “Borrow” button. Just like the Borrow pop-up right now has an option on top for Stable and Variable, Supply too has Stable and Variable options.

Example: Alice, Bob, and Charles.

Alice has MEMESHT and she really believes in it, she’s in it for the long term.

Bob has MEMESHT but not as bullish.

Charles wants to short MEMESHT.

Alice sees MEMESHT under “Assets to supply,” clicks “Supply.”

In pop-up, clicks the “Stable” option on top.

She puts in the amount of MEMESHT tokens of “100.”

She slides the 0% to 100% slider, with 5% increments, to “10%”.

She slides the 6 months to 48 months slider, with 6 month increments, to “12 months.”

Clicks “Submit.”

This means Alice will accept 10% or more borrow rate with a lock of 1 year or more on 100 MEMESHT tokens.

Bob sees MEMESHT under “Assets to supply,” clicks Supply.

In pop-up, clicks the “Stable” option on top.

He puts in the amount of MEMESHT tokens of “200.”

He slides the 0% to 100% slider, with 5% increments, to “50%”.

He slides the 6 months to 48 months slider, with 6 month increments, to “48 months.”

Clicks “Submit.”

This means Bob will accept 50% or more borrow rate with a lock of 4 years (the max) on 200 MEMESHT tokens.

Charles sees ETH under “Assets to supply,” clicks Supply.

In pop-up, clicks the “Stable” option on top.

He puts in the amount of ETH tokens of “0.1.”

He slides the 0% to 100% slider, with 5% increments, to “5%”.

He slides the 6 months to 48 months slider, with 6 month increments, to “48 months.”

Clicks “Submit.”

This means Bob will accept 5% or more borrow rate with a lock of 4 years (the max) on 0.1 ETH tokens.

Charles sees MEMESHT under “Assets to borrow,” clicks Borrow.

In pop-up, clicks the “Stable” option on top.

He sees below the amount field for MEMESHT tokens the text: “Max 300.”

He puts in the amount of MEMESHT tokens of “100.”

The 0% to 100% slider, with 5% increments, defaults the best option of “10%”.

The range between 0% and 10% and 50% and 100% are deactivated.

The 6 months to 48 months slider, with 6 month increments, defaults to the best option of “12 months.”

The range between 6 months and 12 months is deactivated.

He knows he can increase the amounts of rate and lock but has no incentive to.

This means Charles can accept 10% or more borrow rate with a lock of 1 year or more on 100 MEMESHT tokens.

Charles, though, wonders what would happen if he borrowed 200 MEMESHT.

Charles puts in the amount of MEMESHT tokens of “200.”

The 0% to 100% slider, with 5% increments, defaults the best option of “45%”.

Directly below the 45% slider the text: “10% for 12 months, then 50% for 36 months.”

The range between 0% and 45% and 50% and 100% are deactivated.

The 6 months to 48 months slider, with 6 month increments, defaults to the only option of “48 months.”

The range between 6 months and 48 months is deactivated.

He knows he can increase the amounts of rate and lock but has no incentive to.

Clicks “Submit.”

This means Charles will pay 45% or less borrow rate with a lock of 4 years or more on 200 MEMESHT tokens.

Charles knows this because he borrows half of the MEMESHT from Alice at a cheap rate and the rest from Bob at a higher rate. Someone can undercut Alice or Bob, like Daniel might offer 5% for 1 year, and thus his rate would go down, but it can never go over 45%.

EDIT: 11/4/23 - Originally wrote Max 400. 400 is how much he could borrow with his ETH. However, I changed the Max to 300. Because that’s the total amount of liquidity in the books. The max is the minimum of either values.

If he selected 100 MEMESHT, then 0.025 ETH would immediately get locked. But since he selects 200 MEMESHT, the 0.05 ETH gets immediately locked. He won’t get any supply rate from that 0.05 until/unless someone actually borrows from it. The other 0.05 ETH he can withdraw whenever he wants until/unless someone borrows from it. If he repays the 200 MEMESHT with interest, e.g. 290 MEMESHT at any point, then he can withdraw all 0.1 ETH.

Also, the entire order book would be visible on info.aave.com.

1 Like
6

i think this is BS. ppl who have locked in stable rates and have paid premium rate for 12 months will now lose that ability when market recovers and stable coin rates increase.

i think this is very unfair!

like wtf?

can you not fix the “bug”?

this is another example of Aave screwing over actual early adopters

7

You are coming in here and just blame everyone. If you have a better solution, provide it.
And also I am pretty sure it’s better than being hacked and loosing everything. But if you want to get hacked feel free to make a proposal. Seriously never read something stupid like this.

Obviously there is something being worked on in the background but as a first step this feature has to be disabled.

2 Likes
8

Looking forward to the remediation. Well done @bgdlabs @AaveLabs and the bug report submitter :slight_smile:

3 Likes
9

i judge things not by what is said.

it is fine to freeze things but i think immediately jumping to disabling the function should come with more explanation.

it is bug and there is no detail on the bug. yes, they mention that the by describing the bug could be dangerous but is that still valid if they froze the assets?

i find it suspicious that at the exact point where lots of ppls locked in stable rate is lower than market rate, they decide to disable the function.

and i don’t agree with you that the first step is disable.
the first step could be freeze the function until something is resolved.

i have locked in interest rates months ago as i thought interest rates would rise. so for me this is very frustrating. and i would like a proper explanation as i will lose money on this

1 Like
10

The guardian has been activated by the DAO. And in this case I am more than happy they decided to do what they did.
And as stated, more details will follow as soon as it’s “safe”. And I don’t think they will drop stable rates now. Of course not good for people who locked it in. But better than being hacked.

3 Likes
11

yes, i have no problem with freezing assets.

i would just be pissed if they disabled stable rates and i lost my locked in rate just as the market rate is higher than my locked in stable rate.

i am a user and i am more than entitled to express my opinion. this is not just about ppl coming up with ideas but it is also about ppl questioning current proposals.

2 Likes
12

Great job Aave team. Kudos. It might be a good opportunity to make a new round of audits.

2 Likes
13

Hello dear All, during the solving of this issue is it possible to swap the collateral eth v2 for example aweth to adai at 1inch? will it works?

14

If you have locked in a competitive rate, you will keep it. The feature will be disabled for new positions only. The bug is unfortunately in one core mechanism that can be abused and cannot be fixed.

1 Like
15

i don’t see how this is “screwing early adopters”. alternative is a protocol wide hack, you would prefer that maybe? We as protocol devs do the best we can to help the dao understanding how to navigate these kind of issues. If we say the only solution is disable the stable rate, that’s because it is the less risky and in this case only solution to protect the protocol. I am not sure where do you see “screwing early adopters” here and why anyone would want to do that.

1 Like
16

Thanks for the update. Any idea when we will be possible to use borrow and pay loan on aave eth V2?

18

ok thanks. my first response was a bit emotional but my understanding was that they were voting to disable the stable coin rate function after temporarily freezing it. my understanding of disable rate stable was that even if we had locked in the stable rate it would not longer be locked in as they were disabling that functionality.

of course i do not want a protocol wide hack. i just did not understand why after freezing stable pools they were rushing in to disabling the functionality without a proper discussion. as far as i understand the vote to disable the functionality is already going through.

i will be 100% honest i don’t trust DAO’s or devs or any human being online for that matter so when i perceive that i may be “screwed over” i question it.

1 Like
19

@bgdlabs for those that have locked in a stable rate already, will they be able to keep that stable rate?

20

I’m really surprised by your response.

  1. You’re fine, your stable rate will stay the same and you don’t lose it.

  2. If you did lose the stable rate… What would you have preferred… Funds at risk or higher interest rate? I know what I would have chosen…

Good work by the @AaveLabs & @bgdlabs and the guardians (Which they really deserve praise for coming together so fast on a Saturday) for their fast response on a Saturday in solving this.

Overall, another strong showing of Aave DAO’s ability to respond to incidents fast.

3 Likes
21

Hey @chippervan, it’s totally okay to feel that way. Afterall, you have assets at stake and being in the dark about changes like these is stressful.

I think @bgdlabs and the Aave Gaurdian did the right thing here. If funds are at risk, it’s important to mitigate the risk. In this case, it was to pause some markets. While this protects everyone, it creates problems for some. These problems can be lessened by temporarily disabling stable borrowing. This allows a patch to be developed, tested, and audited with causing minimal disruption.

To protect users, details of the exploit can not be published or discussed until after it is resolved. This is the best path forward.

4 Likes
22

i don’t quite understand what you are saying.

apparently a white reported a bug, the team froze assets related to the bug, then team went to vote to disable stable interest rate function, that is where we are now.

i am not sure if those that locked in stable rate will still have that after the stable rate function has been disabled. if not then are there other options?

obviously i don’t want all funds to be at risk.

the team are paid lots of money. i think that is great. but i think it is fair to ask questions