Normally a virtual machine is not offered to upgrade from Windows 10 to Windows 11.
This is a manual is a step by step workaround using a "Windows Server" installer which ignores the fact that the system is a virtual machine.
Go to the Windows Download page and download the "Download Windows 11 Disk Image (ISO) for x64 devices" by selecting "Windows 11 (multi-edition ISO for x64 devices)".
When using VirtualBox add the ISO-file as an optical storage device.
Create a shortcut or use the command-line to execute setup.exe having the product switch for the server.
setup.exe /product server
Start the setup as described in the previous step and even when in says "Install Windows Server" start installing pressing the Next button.
Now it will check for updates and asks "Choose what to Keep" and select "Keep files, settings and apps" and press the Next button.
Now the installer says "Making sure you're ready to install" followed by "Ready to install" and the recap of what is installed and how which is the "Windows 11" flavor of your installed Windows 10 one.
Press the "Install" button and wait... ... ... until the virtual machine reboots.
After reboot run Windows Updates until nothing is updatabled anymore and you are ready to go.