Hello all,
I've just released Dusk v15[1] with its main (sole) focus being the addition of
EFI support. Dusk can boot on EFI v1.x and UEFI v2.x, both on AMD64 and i386
architectures. This includes Intel Macs, both 64-bit and 32-bit. It includes
support for text console (grid too), text input, block I/Os and graphical
output.
You can easily build a EFI boot media by looking at "deploy/efi".
Although the EFI specs are pretty big, all in all, implementing drivers for it
is straightforward. I didn't have many bad surprises when I've tried it on
different hardware. It pretty much just works. No doubt, this is at the cost of
horribly complicated firmware underneath, but still, it works.
Onwards,
Virgil
[1]: https://git.sr.ht/~vdupras/duskos/tree/master/item/CHANGELOG.md#v15---20241227
On 12/27/24 12:55 PM, Virgil Dupras wrote:
> Hello all,>> I've just released Dusk v15[1] with its main (sole) focus being the addition of> EFI support. Dusk can boot on EFI v1.x and UEFI v2.x, both on AMD64 and i386> architectures. This includes Intel Macs, both 64-bit and 32-bit. It includes> support for text console (grid too), text input, block I/Os and graphical> output.>> You can easily build a EFI boot media by looking at "deploy/efi".>> Although the EFI specs are pretty big, all in all, implementing drivers for it> is straightforward. I didn't have many bad surprises when I've tried it on> different hardware. It pretty much just works. No doubt, this is at the cost of> horribly complicated firmware underneath, but still, it works.>> Onwards,> Virgil>> [1]: https://git.sr.ht/~vdupras/duskos/tree/master/item/CHANGELOG.md#v15---20241227
This is amazing stuff, Virgil!
So the UEFI page [1] on the OSDev wiki says this:
> An independent developer may find more value in using UEFI to write
feature-full UEFI applications, rather than viewing UEFI as a temporary
start-up environment to be jettisoned during the boot process. Unlike
legacy bootloaders, which typically interact with BIOS only enough to
bring up the OS, a UEFI application can implement sophisticated behavior
with the help of UEFI. In other words, an independent developer
shouldn't be in a rush to leave "UEFI-land".
Is Dusk staying in UEFI-land, then?
--Aaron
[1]: https://osdev.wiki/wiki/UEFI#How_to_use_UEFI
On Sat, Dec 28, 2024 at 01:11:24PM -0800, Aaron Miller wrote:
> Is Dusk staying in UEFI-land, then?
Yes. This is what DisableWatchdog is about. Otherwise, EFI firmware by default
resets the system after 5 minutes.
That being said, I haven't fully explored yet what this implies. EFI keeps some
services available at runtime. Maybe everything Dusk uses could be used after
ExitBootServices(). Also, maybe that not exiting boot services results in bad
long term effects such as memory leaks or something. To be determined with
usage.
Hello,
Am 28.12.2024 um 22:51 schrieb Virgil Dupras:
> That being said, I haven't fully explored yet what this implies. EFI keeps some> services available at runtime. Maybe everything Dusk uses could be used after> ExitBootServices(). Also, maybe that not exiting boot services results in bad> long term effects such as memory leaks or something. To be determined with> usage.
IIRC, Input devices (keyboard/mouse) and disk devices are no longer
available once boot services have been exited. And for output, the only
thing that remains is a linear framebuffer if GOP provides one.
And with memory leaks I have only been bitten the opposite way - the
Boot services themselves do not leak memory as far as I have seen. With
opposite way I mean: When you launch an application (written yourself)
from EFI shell that leaks memory, then exit back into the UEFI to boot
another (CSM based) OS without rebooting, it can happen that the startup
memory map for CSM boot is "permanently" altered (I wrote about it here:
<https://superuser.com/q/746497/1724>) to avoid having drivers loaded
into the "lost" memory.
UEFI documentation has been updated since then: an EFI application that
intends to exit needs to ensure it frees all of its allocated memory
before doing so.
Regards,
Michael
On Sun, Dec 29, 2024 at 12:56:35AM +0100, Michael Schierl wrote:
> Hello,> > > Am 28.12.2024 um 22:51 schrieb Virgil Dupras:> > > That being said, I haven't fully explored yet what this implies. EFI keeps some> > services available at runtime. Maybe everything Dusk uses could be used after> > ExitBootServices(). Also, maybe that not exiting boot services results in bad> > long term effects such as memory leaks or something. To be determined with> > usage.> > IIRC, Input devices (keyboard/mouse) and disk devices are no longer> available once boot services have been exited. And for output, the only> thing that remains is a linear framebuffer if GOP provides one.> > And with memory leaks I have only been bitten the opposite way - the> Boot services themselves do not leak memory as far as I have seen. With> opposite way I mean: When you launch an application (written yourself)> from EFI shell that leaks memory, then exit back into the UEFI to boot> another (CSM based) OS without rebooting, it can happen that the startup> memory map for CSM boot is "permanently" altered (I wrote about it here:> <https://superuser.com/q/746497/1724>) to avoid having drivers loaded> into the "lost" memory.> > UEFI documentation has been updated since then: an EFI application that> intends to exit needs to ensure it frees all of its allocated memory> before doing so.> > > Regards,> > > Michael
If I understand this thread well, any "double" boot failure would result in
some reserved memory variable being bumped up permanently? That's sick. Broken
by design. I'll keep an eye on that flaw during my EFI experiments[1] as to not
brick myself out of a machine... This also means that a EFI-based "reboot"
command could be dangerous.
To answer your question from the thread about BS_ and RT_, it's BootServices
and RuntimeServices. Those categories correspond to the enum described in
AllocatePages(). As the specs says, most of the time, a EFI application will
want to allocate LoaderData memory type.
Regards,
Virgil
[1]: I also don't bother freeing allocated pages on failure... in fact, at any
time.
Hello Virgil,
Am 29.12.2024 um 14:31 schrieb Virgil Dupras:
> On Sun, Dec 29, 2024 at 12:56:35AM +0100, Michael Schierl wrote:> If I understand this thread well, any "double" boot failure would result in> some reserved memory variable being bumped up permanently?
In some broken UEFI implementations, yes. In fact, only the first boot
needs to fail and throw you back into the OS selection menu (or fall
back to another OS automatically), the second one may succeed. I myself
have only experienced it when the second OS booted used CSM, but I won't
guarantee that. I am aware of later UEFI revisions of the same mainboard
that reset and recalculate the memory map if you reset the UEFI to
factory default, but don't count on that.
OTOH, when you are tech-savvy enough, you probably can manually fix the
relevant UEFI variables, or just never boot the machine in CSM mode again :D
The other situation that is known to brick some UEFIs (to the point they
won't boot any more) is to use more than 50% of EFI variable space. This
also affects only some older devices, when they try to defragment the
variable space, they end up in an infinite loop.
> That's sick. Broken> by design. I'll keep an eye on that flaw during my EFI experiments[1] as to not> brick myself out of a machine... This also means that a EFI-based "reboot"> command could be dangerous.
Reboot (i.e.
<https://uefi.org/specs/UEFI/2.10/08_Services_Runtime_Services.html#reset-system>)
is fine. Only critical paths are
<https://uefi.org/specs/UEFI/2.10/07_Services_Boot_Services.html#efi-boot-services-exit>
or cleanly returning to the caller who called your efi_main function. I
assume you can just avoid doing any of these.
> To answer your question from the thread about BS_ and RT_, it's BootServices> and RuntimeServices. Those categories correspond to the enum described in> AllocatePages(). As the specs says, most of the time, a EFI application will> want to allocate LoaderData memory type.
I don't remember what question you are referring to, sorry.
> [1]: I also don't bother freeing allocated pages on failure... in fact, at any> time.
Then just never exit() :)
Regards,
Michael
Hello Virgil,
Am 27.12.2024 um 21:55 schrieb Virgil Dupras:
> You can easily build a EFI boot media by looking at "deploy/efi".
When you encourage us to test it, of course I do. I only tested it on
two virtual environments, VirtualBox and Microsoft Hyper-V (both on
Windows 11). For Hyper-V, to make this a bit of a challenge, I chose the
"Generation 2 VM" without legacy support, so all you have here is UEFI
and some modern virtio drivers. It is known to be quite picky when
running actual OSes (apart from 64-bit Windows, 64-bit Linux and FreeBSD
most stuff does not work out-of-the box, even other BSDs), but EFI
services should work according to spec.
And it ended with a mere "EFI error 00000007boot failure" - screenshot
at <https://gist.github.com/schierlm/6471b42828ff34f3129475a9fbb9b9f0>.
I know that 7 is device error, but I guess to find out which device did
not work I need to change/instrument the code. A bit disappointing
assuming you have such a "horribly complicated" firmware underneath and
the information I get from the error is a mere 32 bits (reminds me of
hardware POST code cards...).
Let's try VirtualBox: That one works a lot better. Oberon does not
start, though, as there is no drv/timer support.
<https://uefi.org/specs/UEFI/2.9_A/38_Micellaneous_Protocols.html#efi-timestamp-protocol>
may help here.
And it pegs my CPU at 100% (and my host's fans at full speed). Maybe
does not hurt too much on old hardware, but modern hardware (maybe
notebooks on battery) really don't like that too much.
For an idle loop, you probably need
<https://uefi.org/specs/UEFI/2.9_A/07_Services_Boot_Services.html#efi-boot-services-createevent>
to create an EVT_TIMER.
<https://uefi.org/specs/UEFI/2.9_A/07_Services_Boot_Services.html#efi-boot-services-settimer>
to set a relative (one-shot) trigger time in the future.
<https://uefi.org/specs/UEFI/2.9_A/07_Services_Boot_Services.html#efi-boot-services-waitforevent>
To do the actual waiting. In the event set, you probably want to add
your timer event along with the event from the text input protocol
(keyboard) and the event from the pointer input protocol (mouse).
Regards,
Michael
On Tue, Dec 31, 2024 at 11:28:57PM +0100, Michael Schierl wrote:
> Hello Virgil,> > > Am 27.12.2024 um 21:55 schrieb Virgil Dupras:> > > You can easily build a EFI boot media by looking at "deploy/efi".> > When you encourage us to test it, of course I do. I only tested it on> two virtual environments, VirtualBox and Microsoft Hyper-V (both on> Windows 11). For Hyper-V, to make this a bit of a challenge, I chose the> "Generation 2 VM" without legacy support, so all you have here is UEFI> and some modern virtio drivers. It is known to be quite picky when> running actual OSes (apart from 64-bit Windows, 64-bit Linux and FreeBSD> most stuff does not work out-of-the box, even other BSDs), but EFI> services should work according to spec.> > And it ended with a mere "EFI error 00000007boot failure" - screenshot> at <https://gist.github.com/schierlm/6471b42828ff34f3129475a9fbb9b9f0>.> I know that 7 is device error, but I guess to find out which device did> not work I need to change/instrument the code. A bit disappointing> assuming you have such a "horribly complicated" firmware underneath and> the information I get from the error is a mere 32 bits (reminds me of> hardware POST code cards...).> > > Let's try VirtualBox: That one works a lot better. Oberon does not> start, though, as there is no drv/timer support.> <https://uefi.org/specs/UEFI/2.9_A/38_Micellaneous_Protocols.html#efi-timestamp-protocol>> may help here.> > And it pegs my CPU at 100% (and my host's fans at full speed). Maybe> does not hurt too much on old hardware, but modern hardware (maybe> notebooks on battery) really don't like that too much.> > For an idle loop, you probably need> > <https://uefi.org/specs/UEFI/2.9_A/07_Services_Boot_Services.html#efi-boot-services-createevent>> to create an EVT_TIMER.> > <https://uefi.org/specs/UEFI/2.9_A/07_Services_Boot_Services.html#efi-boot-services-settimer>> to set a relative (one-shot) trigger time in the future.> > <https://uefi.org/specs/UEFI/2.9_A/07_Services_Boot_Services.html#efi-boot-services-waitforevent>> To do the actual waiting. In the event set, you probably want to add> your timer event along with the event from the text input protocol> (keyboard) and the event from the pointer input protocol (mouse).> > > Regards,> > > Michael
Thanks for testing. I don't have access to this program so my debugging
abilities are limited, but it's good to know that it fails somewhere.
From the screenshot, it looks like failure happened before init.fs could be
loaded, so it means it's drv/efi/blkio who's likely at fault.
As for the rest, yeah, "running on EFI" is a bit of an overstatement. It's still
missing the timer, the mouse, and smoothing out rough edges. Passing the whole
test suite is such a big event, I get carried away sometimes :)
Thanks for the pointers. For the idle loop, Stall looks enticing to me for its
straightforwardness. Yes, the correct way to idle is, as you hint, using the
event mechanism and have a list of all possible I/O events. But waiting for a
few microseconds and polling I/O should do the trick too, albeit with spurious
polling. But at least it won't have the CPU spin 100%.
Regards,
Virgil
Hello Virgil,
Am 01.01.2025 um 14:01 schrieb Virgil Dupras:
> Thanks for testing. I don't have access to this program so my debugging> abilities are limited, but it's good to know that it fails somewhere.>> From the screenshot, it looks like failure happened before init.fs could be> loaded, so it means it's drv/efi/blkio who's likely at fault.
It's
<https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.html#efi-block-io-protocol-reset>.
Calling it with ExtendedVerification=1 results in EFI_DEVICE_ERROR.
Replacing 1 by 0 makes it boot flawlessly.
> As for the rest, yeah, "running on EFI" is a bit of an overstatement. It's still> missing the timer, the mouse, and smoothing out rough edges. Passing the whole> test suite is such a big event, I get carried away sometimes :)
Running all the manual tests show that keyboard support is also
suboptimal. For retrieving e.g. shifted arrow keys, ReadKeyStrokeEx is
needed, which is optional and requires fallback code. But even with
ReadKeyStroke you can receive normal arrow keys, function keys and
Escape key, which your current code does not handle.
> Thanks for the pointers. For the idle loop, Stall looks enticing to me for its> straightforwardness.
I just checked with Dusk OS in all VM software I have, Stall always does
a busy wait (i.e. spins the CPU at 100%). Probably makes sens since
Stall can be used in drivers and is intended to maintain protocol delays.
Yes, the correct way to idle is, as you hint, using the
> event mechanism and have a list of all possible I/O events. But waiting for a> few microseconds and polling I/O should do the trick too, albeit with spurious> polling. But at least it won't have the CPU spin 100%.
Yes, if you just use the timer and wait for it, instead of calling
Stall, it should be sufficient. But adding "only" keyboard (and maybe
mouse) events to it does not increase the complexity by much, but
improves the responsiveness.
About microseconds: If I recall correctly, the TianoCore EDK reference
implementation has two different event wait implementation stubs, one
for times of at most 1 millisecond and one for times longer than 1
millisecond. While I have no idea which actual hardware implementations
actually implement them differently (maybe test it in QEMU if you like),
it may be a good idea to wait for, say, 10 milliseconds (or more). I
know this is also "a few microseconds" (just 10000 of them), but
probably you intended a shorter interval when saying "a few
microseconds". OTOH, the operator can tweak the interval in case he is
low on battery power.
Regards,
Michael
On Wed, Jan 01, 2025 at 07:23:25PM +0100, Michael Schierl wrote:
> Hello Virgil,> > > Am 01.01.2025 um 14:01 schrieb Virgil Dupras:> > > Thanks for testing. I don't have access to this program so my debugging> > abilities are limited, but it's good to know that it fails somewhere.> > > > From the screenshot, it looks like failure happened before init.fs could be> > loaded, so it means it's drv/efi/blkio who's likely at fault.> > It's> <https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.html#efi-block-io-protocol-reset>.> Calling it with ExtendedVerification=1 results in EFI_DEVICE_ERROR.> Replacing 1 by 0 makes it boot flawlessly.
Oh, it's tricky. I set this flag to 1 after having observed on some hardware (I
think it was my Mac Mini) that having it to 0 prevented ReadBlocks from working
afterwards. Trying both could do the trick. I'd have to begin with
ExtendedVerification=1 because in my experiments, it's not Reset that failed,
but subsequent ReadBlocks. But if Reset fails with ExtendedVerification=1, then
I retry with 0, then fail if there's still an error.
But then again, this was at a time where I was trying a lot of things to get
the Mac port going. Maybe I should try ExtendedVerification=0 on all hardware
again...
> > As for the rest, yeah, "running on EFI" is a bit of an overstatement. It's still> > missing the timer, the mouse, and smoothing out rough edges. Passing the whole> > test suite is such a big event, I get carried away sometimes :)> > Running all the manual tests show that keyboard support is also> suboptimal. For retrieving e.g. shifted arrow keys, ReadKeyStrokeEx is> needed, which is optional and requires fallback code. But even with> ReadKeyStroke you can receive normal arrow keys, function keys and> Escape key, which your current code does not handle.
Yes I've also noticed this this morning. On my short term TODO list.
Unfortunately, the "Ex" family of SIMPLE_INPUT is only available on UEFI, not
EFI, so this will have to be an optional separate driver.
> > Thanks for the pointers. For the idle loop, Stall looks enticing to me for its> > straightforwardness.> > I just checked with Dusk OS in all VM software I have, Stall always does> a busy wait (i.e. spins the CPU at 100%). Probably makes sens since> Stall can be used in drivers and is intended to maintain protocol delays.> > > Yes, the correct way to idle is, as you hint, using the> > event mechanism and have a list of all possible I/O events. But waiting for a> > few microseconds and polling I/O should do the trick too, albeit with spurious> > polling. But at least it won't have the CPU spin 100%.> > Yes, if you just use the timer and wait for it, instead of calling> Stall, it should be sufficient. But adding "only" keyboard (and maybe> mouse) events to it does not increase the complexity by much, but> improves the responsiveness.> > About microseconds: If I recall correctly, the TianoCore EDK reference> implementation has two different event wait implementation stubs, one> for times of at most 1 millisecond and one for times longer than 1> millisecond. While I have no idea which actual hardware implementations> actually implement them differently (maybe test it in QEMU if you like),> it may be a good idea to wait for, say, 10 milliseconds (or more). I> know this is also "a few microseconds" (just 10000 of them), but> probably you intended a shorter interval when saying "a few> microseconds". OTOH, the operator can tweak the interval in case he is> low on battery power.
You're right, I saw the whole thing as more complex than it was. In my mind,
waiting after kbd+mouse I/O precluded also waiting for a fixed amount of time.
That would have been problematic/buggy. But the way EFI events are done, it
seems that we can very well do "kbd OR mouse OR 10ms", which is the best of
all worlds.
Thanks for your input.
Regards,
Virgil
> You're right, I saw the whole thing as more complex than it was. In my mind,> waiting after kbd+mouse I/O precluded also waiting for a fixed amount of time.> That would have been problematic/buggy. But the way EFI events are done, it> seems that we can very well do "kbd OR mouse OR 10ms", which is the best of> all worlds.
Noob question: if you can idle by waiting for the keyboard and mouse, why
would you want to also idle with a timer?
On Thu, Jan 02, 2025 at 01:25:56PM -0500, Polifemo wrote:
> > You're right, I saw the whole thing as more complex than it was. In my mind,> > waiting after kbd+mouse I/O precluded also waiting for a fixed amount of time.> > That would have been problematic/buggy. But the way EFI events are done, it> > seems that we can very well do "kbd OR mouse OR 10ms", which is the best of> > all worlds.> > Noob question: if you can idle by waiting for the keyboard and mouse, why> would you want to also idle with a timer?
Any other kind of I/O: USB transfer, Serial communication, etc. That was the
cause of my initial reserve with the event wait scheme. You have to
exhaustively list the possible events you want to wait for or you run the risk
of over-idling and missing I/O windows.
If some of your I/O are not EFI based, things would get even more complicated.
But if you do "kbd OR mouse OR small amount of time", then the idling never
gets stuck.
Hello Virgil,
Am 01.01.2025 um 20:49 schrieb Virgil Dupras:
>> Running all the manual tests show that keyboard support is also>> suboptimal. For retrieving e.g. shifted arrow keys, ReadKeyStrokeEx is>> needed, which is optional and requires fallback code. But even with>> ReadKeyStroke you can receive normal arrow keys, function keys and>> Escape key, which your current code does not handle.>> Yes I've also noticed this this morning. On my short term TODO list.> Unfortunately, the "Ex" family of SIMPLE_INPUT is only available on UEFI, not> EFI, so this will have to be an optional separate driver.
Refering to
<https://git.sr.ht/~vdupras/duskos/commit/25b1003d160c88252a4daf6260c6e64a0c090aa9#fs/doc/drv/efi/kbdex.txt-1-7>,
I'm curious if there was any real-world hardware out there you tested it
on where passing EFI_BOOT_SERVICES.ConsoleInHandle as kbdid did not
result in the expected default (i.e. using the same keyboard that is
also used for the non-extended keyboard read via
EFI_BOOT_SERVICES.ConIn). Because I've not seen that, but I don't have
as much old hardware as you do :)
Having an option to access different keyboards is definitely fancy, but
I assume most people prefer if you default to the default one (pun
intended) instead of making the operator read device names and guess
which one is the one you are currently using.
Regards,
Michael
On Sun, Jan 05, 2025 at 01:14:46AM +0100, Michael Schierl wrote:
> Hello Virgil,> > > Am 01.01.2025 um 20:49 schrieb Virgil Dupras:> > > Running all the manual tests show that keyboard support is also> > > suboptimal. For retrieving e.g. shifted arrow keys, ReadKeyStrokeEx is> > > needed, which is optional and requires fallback code. But even with> > > ReadKeyStroke you can receive normal arrow keys, function keys and> > > Escape key, which your current code does not handle.> > > > Yes I've also noticed this this morning. On my short term TODO list.> > Unfortunately, the "Ex" family of SIMPLE_INPUT is only available on UEFI, not> > EFI, so this will have to be an optional separate driver.> > Refering to> <https://git.sr.ht/~vdupras/duskos/commit/25b1003d160c88252a4daf6260c6e64a0c090aa9#fs/doc/drv/efi/kbdex.txt-1-7>,> I'm curious if there was any real-world hardware out there you tested it> on where passing EFI_BOOT_SERVICES.ConsoleInHandle as kbdid did not> result in the expected default (i.e. using the same keyboard that is> also used for the non-extended keyboard read via> EFI_BOOT_SERVICES.ConIn). Because I've not seen that, but I don't have> as much old hardware as you do :)> > Having an option to access different keyboards is definitely fancy, but> I assume most people prefer if you default to the default one (pun> intended) instead of making the operator read device names and guess> which one is the one you are currently using.> > > Regards,> > > Michael
Hello Michael,
It didn't even occur to me to try it! Of course, it's the correct way. Thanks
for keeping me on the right track :)
Regards,
Virgil