TL;DR:
- This post was an attempt to discredit Chaos and ensure vendor-lock in from a well-connected competitor
- The critique misrepresented a proof-of-concept video with attacks clearly stated in the accompanying disclaimers, but we have explained the relevant methodology accordingly
- We want to shift the entire risk of this engagement onto our ability to deliver, so we will work for FREE for the first 6 months of this engagement.
I have read this post multiple times and have seriously weighed the appropriate response. Tarun’s response is malicious, misleading, unprecedented, and unnecessary. It requires a clear, direct response. Tarun’s goal is to discredit Chaos and insinuate that we are unqualified to provide services to the DAO. This wasn’t only an attack on me, the CEO, but the entire Chaos team who has worked tirelessly to build a scalable platform to best support the DAOs who put their trust in us. I felt the need to respond in kind to put my support behind the 20+ people working at Chaos to make DeFi more robust and better secure individual user assets.
In this response, I will share why Tarun’s response is clearly incorrect, why Chaos Labs can and will deliver world-class services to the Aave DAO, then refute claims lobbed at us, and propose a revised path forward.
To get to the brass tacks: our initial proposal has been up for nearly 30 days without a single response or question from Gauntlet. During that time, we’ve received numerous messages that Gauntlet has been lobbying against us to prevent us from servicing the DAO as their potential contract renewal date approaches. While it was disheartening to hear about a competitor playing dirty and relying on clout/relationships to advance their business agenda, we were happy to focus on deliverables: showcasing the platform’s ability and ultimately letting the quality of our product and work speak for itself.
While many DAOs we speak with are concerned with “vendor lock-in” when choosing partners, they mostly look at it from a technical perspective. The back-door lobbying and gate-keeping from early DAO service providers is a far more concerning aspect of this entanglement and one we are now watching play out. These early participants, like Gauntlet, have the sufficient cachet to suffocate any potential competitors away from the DAO and prevent it from receiving competing services.
As an early member and contributor of this protocol, Gauntlet has both the connections and the token allocation to make a significant impact on how the community interprets major decisions. We are happy to engage on the merits of the proposal or those of the simulation platform, but not get into mudslinging across a proof of concept prior to engagement.
We also understand that stakeholders and community members may not be up to date with the inside baseball and tactics that Gauntlet deploys, so we’ll take the time to explicitly provide a pedagogical understanding of why Gauntlet claims are unbounded in truth, inappropriate, and just plain wrong.
Technical responses
The heart of the matter is that you’re attacking a single simulation that was built as a demonstration for the Aave community. Creating high-fidelity simulations and models can take weeks or months. You need to look no further than Gauntlet’s outline for v3 support, in which it references the “several months” it has spent to expand risk modeling to support v3.
In addition to the financial resources needed, training predictive models take time and engineering hours. This simulation is decidedly smaller in scope as a proof of concept as the real thing costs resources we can’t spare before the guarantee of an engagement, as should be expected from any vendor.
There’s a clear reason why we build a smaller scoped proof of concept. Running simulations is a resource-intensive task. The compute cycles involved in running them are not free. The time engineers spend running them is not free. The purpose of these POCs is to illustrate what the platform is capable of, not to stress test it. When you go to buy a car, you can take it for a test drive. You can see how fast it goes from 0 to 60, but you don’t get to run it for 300k miles.
The POC clearly demonstrates what our product can do, and mass-scale simulations will be run once there is a contract in place to handle the material operating expense involved. Suggesting that any service provider approaching the DAO should only do so after incurring an incredibly high expense is to introduce a form of bureaucratic capture and incumbency bias not dissimilar from what we see when big banks or big tech companies lobby congress to regulate their industry.
The work we have done for Aave, beyond the Risk Dashboard for which we received a small grant, has been well received. We built dashboards at half the price other grantees received for similar tasks. We spent days debugging Aave subgraphs, finding critical errors, and shipping PRs to benefit the community. We have been working with the community in good faith to show progress, create walkthrough educational materials, and demonstrate how our platform can be utilized, but at a fraction of the cost.
The fact that you are attacking and analyzing a proof-of-concept video is misleading.
Let’s clarify the purpose of the PoC and then address the claims lobbed at us:
First of all - the goals of the stETH:ETH depeg simulation were to:
- Show the capabilities of the platform, both in how it interprets on-chain data and the transparency-focused tools built to allow for community analysis of the underlying simulation.
- Highlight the minimized need for contract/protocol reconstruction and assumptions by not copying the functionality into a different domain-specific language.
At no point did we claim that the results of a single simulation are statistically significant. As such, all assumptions were clearly stated in the video and blog post.
Now let’s discuss the accusations.
Let’s begin by lumping all the attempts to confuse readers into a single bucket:
-
“The assumptions are wholly unrealistic:
1. “In this simulation, we ignore the effect of stETH de-peg on other asset prices.” This is completely not true in practice and moreover, the liquidity and/or slippage curves of other assets are also correlated to stETH/ETH!
2. “We do not simulate any stETH buy pressure on Curve in order to speed up the cascading liquidation effect.” This was in fact what ensured that Aave was safe — the original posters clearly did not look at on-chain data during the large liquidation events.
-
“Choosing a stopping time for a simulation implicitly impacts the statistical quality of the results.”
-
“For reference, Gauntlet runs simulations for a minimum of 1 day, which simulates 5760 blocks and we run over 40,000 simulations per day sampling different statistical configurations.”
Again - It’s a proof-of-concept, not a case study or report. This is why they are stated as disclaimers and assumptions on the blog and video walkthroughs. Therefore, all these arguments are baseless.
Now I will address several points on which we feel the community deserves clarity.
-
“The authors depeg stETH to a large deviation and measure whether liquidations occur as a metric of protocol health. This misses a few major statistical observations that need to be constantly retrained based on off-chain and on-chain data”:
-
“This model seems to ignore this effect and the volatility (or any higher order moments of the price process).”
In the demo simulation, we examine an edge case of stETH de-peg at a time of block 15006921. At that time, stETH liquidity was primarily on Curve. We clearly address that fact, explaining the reasoning for our liquidity model:
“Unlike most major crypto assets, where significant liquidity is found on CeFi venues, stETH liquidity is mainly provided on Curve. This allows us to simulate inter-protocol dependencies and effects with minimal off-chain assumptions, accelerating cascading liquidations.” (Sources: Link, Link)
The model does not ignore arbitragers’ effect on price stabilization but utilizes the fact the stETH liquidity was mostly on Curve at the time of recording. Therefore, this claim is, again, meaningless. It is important to note that this was a conscious decision, not a technical limitation. We’ve modeled a variety of arbitragers into the platform for different simulations and will be open-sourcing their code for the community to review as we engage with Aave and release the parameter optimization platform publicly. Similar to other open-source development (i.e. subgraphs), it is our hope that multiple community members review these and help optimize them both for Chaos’ simulations and individual developer testing.
-
“There is no description of the liquidity model used.”
Chaos Labs simulations utilize a mainnet fork as our simulation runtime environment. As such, our agents are interacting with the Curve contracts as they would on mainnet without having to make assumptions on the liquidity model. Since the majority of stETH was on Curve during the recording, we believe this is the most accurate liquidity/market impact model for that simulation. Because of this, we can estimate on-chain liquidators’ profits more accurately.
-
“The impact of such a shock on the system depends on volatility conditions throughout DeFi — the smooth curves rendered in the case study assume purely deterministic behavior which is very much not what you see on-chain or in the mempool.”
The impact of arbitrageurs is addressed. We do make an assumption of rational, efficient liquidation bots. We believe that this is a fair assumption since although their execution time and order are not deterministic, their behavior is predictable as rational agents.
-
“Choosing a stopping time for a simulation implicitly impacts the statistical quality of the results.”
Nowhere in the proof-of-concept is there an assumption of infinite ****CEX liquidity. On the contrary, the lack of CEX liquidity provides the reasoning for looking at Curve Pool liquidity as the primary liquidity model. That fact enhances the effect of cascading liquidations, as shown in that simulation, and it is the reason we chose that edge case.
-
Team Background
As I mentioned, this post was not just an attack on my competency, but on the entire team at Chaos Labs. A team of hard-working people who want to see Aave and DeFi as a whole succeed. We’re not scared of DAO politics or incumbent bullying. We want to make sure that the community knows what it’s getting when engaging with us.
We are aligned with Gauntlet that the appropriate methodology to determine optimal parameters for DeFi protocols is Monte Carlo simulations at scale, but approach it from a different perspective. Our team has years of experience in building data-driven simulations, determining billion-dollar outcomes across FAANG companies, and matters of national security. I’ve personally led statistical experiments and analyses across internet.org, Instagram, and Facebook. Our team’s past experience includes building simulation software for calculating incoming missile trajectories and diagnosing and predicting medical diseases. We are well equipped to handle the challenges facing the DAO (and building these simulations at scale) and are eager to prove it throughout this engagement.
Proposal Pricing Update
As we clearly stated above - we will let our product and work speak for themselves and we are updating the payment structure to even further demonstrate this. We want to shift the entire risk of this engagement onto our ability to deliver.
We will work for FREE for the first 6 months of this engagement. Chaos does not get paid until after the 6-month anniversary when the DAO has the ability to terminate the contract. That means that if the DAO terminates, it will have paid Chaos $0 for work done. We will provide a detailed report on our progress that will give the community sufficient information to make that decision.
Compensation Model:
- $500,000 flat engagement fee paid in USDC streamed linearly starting at the 6-month anniversary of the public vote and streamed over the remainder of the contract
- Incentives based on delivery, payable no earlier than 6-month anniversary and based on a trailing 7-day TWAP:
-
$175,000 paid in AAVE based on delivery of the Aave Asset Listing Portal
-
$175,000 paid in AAVE based on delivery of the Aave Parameter Recommendations Tools
(Delivery is defined by open access of the tool to the community and shared in the Aave Forum)
-
Conflict of Interest
We would like to highlight to the Aave community the inherent conflict of interest present between all parties involved here (Chaos, Gauntlet, Standard Crypto, and each side’s relevant investors) regarding the onboarding of Chaos Labs as a contributor to Aave. This proposal does not replace Gauntlet and we hope to have a productive relationship will all Aave contributors. With that said, we acknowledge that we are 100% competitive with each other in hopes of providing the best product and service to the protocol at the most attractive price. With token holder approval, Chaos Labs will prove that there are other methods — ones more transparent, efficient, and scalable — to properly protect Aave and that we are committed to helping retain treasury funds (and hopefully increase them) through the current market cycle.
Due to this conflict, Chaos Labs will not vote on this proposal and has requested its investors to do the same. We wish to allow the Aave community to independently determine the value of bringing us on board with the revised terms.
Conclusion
The amount of AUM at stake affects the magnitude and seriousness of the situation, but not the complexity. At the end of the day, this is an optimization problem that uses probabilistic predictions to find a balance between capital efficiency and economic security. The Chaos team is experienced and has a long history of working on complex data-rich, security-oriented issues in both the public and private sectors. We wrote the proposal to communicate it to the community in terms that are digestible to stakeholders and not to overcomplicate it with unnecessary technical jargon (which we have expanded upon above). In short, analysis paralysis is not the path forward.
The way we like to do business at Chaos is simple and honest. Ultimately, what we propose to do is straightforward. We’re offering a product and service that Aave needs at a fraction of the cost it’s paying for now.
We can go back and forth with an academic debate on simulation optimization models but would prefer to build first and debate once the community has access to our platform, models, agents, and results.
Fundamentally we believe that the long-term success of the DAO model is predicated on a diverse ecosystem of contributors and vendors. Such an ecosystem should create healthy competition in product quality and contract terms ultimately benefiting the DAO and maximizing the ROI. The goal of this revision is to remove any concerns of working with Chaos and make the decision to engage as easy as possible with no additional burden on the treasury required before delivering.