This is great news for folks that use microVMs - "we only use AWS" has been an issue for our stuff (slicer services/sandboxes/actuated self-hosted GitHub runners)
If anyone here can't wait (as it looks like there's very little info on this at the moment..)
I wrote up detailed instructions for Ant Group's KVM-PVM patches. Performance is OK for background servers/tasks, but does take a hit up to 50% on complex builds like Kernels or Go with the K8s client.
DIY/detailed option:
https://blog.alexellis.io/how-to-run-firecracker-without-kvm...
Fully working, pre-built host and guest kernel and rootfs:
https://docs.slicervm.com/tasks/pvm/
I'll definitely be testing this and comparing as soon as it's available. Hopefully it'll be accelerated somewhat compared to the PVM approach. There's still no sign whether those patches will ever end up merged upstream in the Linux Kernel. If you know differently, I'd appreciate a link.
Azure, OCI, DigitalOcean, GCE all support nested virt as an option and do all take a bit of a hit, but it makes for very easy testing / exploration. Bare-metal on Hetzner now has a setup fee of up to 350 EUR.. you can find some stuff with 0 setup fee, but it's usually quite old kit.
Edit: this doesn't look quite as good as the headline.. Options for instances look a bit limited. Someone found some more info here: https://x.com/nanovms/status/2022141660143165598/photo/1
This is a big deal because you can now run Firecracker/other microVMs in an AWS VM instead of expensive AWS bare-metal instances.
GCP has had nested virtualization for a while.
Is nested VMX virtualization in the Linux kernel really that stable?
The technical details are a lot more complex than most realize.
Single level VMX virtualization is relatively straightforward even if there are a lot of details to juggle with VMCS setup and handing exits.
Nested virtualization is a whole another animal as one now also has to handle not just the levels but many things the hardware normally does, plus juggling internal state during transitions between levels.
The LKML is filled with discussions and debates where very sharp contributors are trying to make sense of how it would work.
Amazon turning the feature on is one thing. It working 100% perfectly is quite another…
> Nested virtualization is supported only on 8th generation Intel-based instance types (c8i, m8i, r8i, and their flex variants). When nested virtualization is enabled, Virtual Secure Mode (VSM) is automatically disabled for the instance.
Support for nested virtualization has been added to the main SDKs. In the us-west-2 region, you can already see the "Nested Virtualization" option and use it with the new M8id, C8id, and R8id instance types.
This is really big news for micro-VM sandbox solutions like E2B, which I work on.
This will make it easier to run automated tests in the Android emulator in CI using regular runners. It was a pain dealing with bare-metal instances just for that.
When will AWS add a statement about being bound to professional secrecy (e.g s203 in Germany) so we use the LLM endpoints for sensitive industries https://repost.aws/es/questions/QUOuFPk9TLSUuClI_wYNmVCQ/ser...
I wonder if this is connected to Azure launching OpenShift Virtualization on "Boost" SKUs? There are a lot of VMWare customers going to OpenShift Virt, and apparently the CPU/memory overhead on Azure maxes out around 10% under full load... but then hyper V has been doing a lot of work on it. No idea if nitro includes any of the KVM-on-KVM passthrough of full KVM, to give it an edge here.
Could someone explain why this is might be a big deal?
I remember playing with nested virty some years ago and deciding it is a backwards step except for PoC and the like. Given I haven't personally run out of virty gear, I never needed to do a PoC.
welcome AWS to 2018!
This a great news, but is there any more information about this other than an aws sdk commit? Is this generally available?
Would love to see performance numbers with nested virtualization, particularly that of IO-bound workloads.
Is this only when using the Go SDK?
What's the performance impact for nested virtualization in general? I'd think this would be adding multiple layers of MMU overhead.
I wonder if this will extend SEV-SNP and TDX to the child VMs?
Yo dawg, I heard you like virtualisation so we put virtual servers inside of your virtual servers.
But I'm sure their ToS doesn't allow you to run your own cloud platform inside AWS.
Would these thing be good for openclaw, agents?
Proof that we’re living in a simulation.
How can this be replicated on prem?
Sounds expensive for legacy apps
It also makes me wonder how many other things I might not know that people are trying to do with cloud platforms that aren’t supported by them but have a negligible performance hit for many use cases.
I wonder if providers like Hetzner and Digital Ocean etc. will get this someday also.
"* *Feature*: Launching nested virtualization. This feature allows you to run nested VMs inside virtual (non-bare metal) EC2 instances."
obligatory: https://www.destroyallsoftware.com/talks/the-birth-and-death...
spoiler though: I'm referencing the part where gimp is running in Wine running in asm.js in a Chrome browser running in another asm.js in Firefox
Digital Ocean has always supported nested virtualization.
hell yes, finally
Only took them 9 years. AWS so much innovation.
Remember, “customer obsession”.
But “protect revenue first”.
I feel vindicated :). We put in a lot of effort with great customers to get nested virtualization running well on GCE years ago, and I'm glad to hear AWS is coming around.
You can tell people to just do something else, there's probably a separate natural solution, etc. but sometimes you're willing to sacrifice some peak performance just have that uniformity of operations and control.