I was reading the `fontconfig` source recently (to help me understand what, exactly, is the difference between the various `fc-*` tools and their options) and some of the code scared me.
I thought, "for sure this is a use-after-free" ... but it happened to be safe because some function performed an undocumented incref, and the return value of that function was kept alive.
So this is definitely a high priority thing to port!
... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
> ... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
To my understanding, there's no way to do this: Fil-C is basically a managed runtime for native code, so the FFI implications are similar to those for Go or any other intrusive managed runtime. Which is to say that Fil-C could offer an FFI, but not without losing the blanket exit-instead-of-memory-unsafety guarantees it aims to offer.
What is the actual catch with Fil-C? Memory safety for C based projects without a complete rewrite sounds like a very good idea, but there must be some pitfalls in this approach which this would not work in some cases.
In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
- Not ABI compatible with Yolo-C. So, you have to recompile the whole stack. This is both a major catch, and it is both a bug and a feature. It's a bug for obvious reasons. It's a feature because it means I'll probably be the first to have a totally usable, totally memory safe desktop environment.
- In my testing, it's between 1.2x and 4x slower than Yolo-C. It uses between 2x and 3x more memory. Others have observed higher overheads in certain tests (I've heard of some things being 8x slower). How much this matters depends on your perspective. Imagine running your desktop environment on a 4x slower computer with 3x less memory. You've probably done exactly this and you probably survived the experience. So the catch is: Fil-C is for folks who want the security benefits badly enough.
It's not obvious that either of these catches are fundamental. The Fil-C development philosophy is very much to get something working first, which means downscoping. That's why I didn't include Yolo-C ABI compat as a goal at all and it's also why I haven't done as many optimizations as I should have. If you look at my GitHub you'll see >20 open issues with performance optimization ideas.
> In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
Of course this has happened.
Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C). Fil-C doesn't have an `unsafe` statement and the dependencies are all recompiled with Fil-C.
> Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Have you considered making... Fil-Zig? Would it be easier to do because Zig is already fairly far ahead of C in terms of safety?
> Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C).
To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe", it's just that PAM isn't any safer (because it's unmodified C). Also you can use Rust without libc - https://github.com/sunfishcode/mustang - unfortunately it doesn't seem to have gained much traction which I think is a shame because glibc is a curse.
Most Rust programs actually have very few C dependencies. The big exceptions are libc and OpenSSL, but I think they'll be excised eventually.
Almost all uses of unsafe in Rust are either for FFI, or to avoid the overhead of things like ref-counting. If you use, eg, Rc RefCell everywhere you will achieve safety without issue.
> Almost all uses of unsafe in Rust are either for FFI
Which makes Rust less safe than Fil-C.
Consider that you have a dependency like libc, PAM, openssl, or the like.
In Rust: you will `unsafe` call into the unsafe versions of those libraries. Hence, even if your Rust code is safe in isolation, your program is not safe overall because you're pulling in unsafe code.
In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
> In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
Great work on Fil-C by the way. Very technically impressive stuff!
It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this). More and more C libraries are getting pure rust ports these days. It’s increasingly rare to actually need to pull in C libraries at all. Sticking to safe rust is a bit inconvenient - but I suspect so is using fil-c. I suppose with fil-C you can use any libraries you want, you’ve just gotta recompile them. That’s cool!
Rust has a bunch of unsafe code in std, so you’re still at risk of a bug there causing safety problems. But the same is true of Fil-C.
I could definitely see myself reaching for Fil-C when running legacy code. It’d be nice if there was a way to use fil-C to protect my rust program from memory bugs in C dependencies. But I can think of dozens of ways why that would be tricky to pull off.
Also I’m curious - can Fil-C protect against threading bugs like rust can? Rust has Send & Sync traits to mark which objects can be safely shared between threads. I can’t think of a way to do that in an automated way without help from the programmer.
> It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
Say you wanted to write a tool like sudo in C# or Go. Here's what would happen: you'd end up having to rely on dependent libraries, like pam, which then relies on other things like libselinux and libaudit. And libselinux depends on pcre2. Those things would be compiled with Yolo-C, so while your sudo tool's new code written by you would be safe, the process would mostly contain unsafe code.
But if you write sudo in Fil-C, then you can rely on all of those dependencies being compiled with Fil-C. The whole process is safe.
So, the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
> If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this).
That's impractical for the reasons above.
> Also I’m curious - can Fil-C protect against threading bugs like rust can?
Not like Rust can, but yes, it does.
Fil-C preserves memory safety under races. This means that debugging multithreaded C/C++ programs in Fil-C is a much nicer experience than in Yolo-C. If you do something wrong, you're going to get a panic - similar to how multithreading bugs in Java or C# lead to exceptions.
But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
> the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
Right; that makes a lot of sense. So currently the only implementation of pam is written in Yolo-C. If you want to build a safe, secure tool which depends on pam - like sudo - then you either need to trust pam (like sudo-rs does). Or you need some way to compile pam to safe code. Fil-C is a way to do that.
Just to name them - I can think of two other solutions to this problem:
1. Sandbox pam within a wasm container. This is probably less convenient than Fil-C (although it would allow that wasm container to be linked from yolo-c, rust, go, js, etc). For pam in particular, this would also be much less safe than Fil-C, since memory bugs in pam would still be possible, and might still result in privilege escalation. Wasm only sandboxes memory at the boundary. It wouldn't protect pam itself from its own bugs like Fil-C does.
2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
I think this also reaffirms the role I see for Fil-C: as a tool to make legacy C code (like pam) safe, at the cost of performance. For new code, I'd rather pick a fully GC language if I don't care about performance, or use rust if performance matters. Again, it would be nice if I could have the best of both worlds - use fil-c to wrap and sandbox pam, but write the rest of my program in rust or C# or Go or something else that is already safe without needing to pay fil-c's cost.
> But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
It depends what you're doing, but this can be a pretty big deal. I suppose this puts Fil-C in about the same safety and performance camp as Go, Java and C#. Ie, it has some protections that rust is missing - like there aren't any unsafe blocks. And its missing some protections rust has - like better thread safety. The unique selling point is it works with existing C code, like old C libraries that we don't want to rewrite. Thanks for your work on it!
> 2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
OK, let's think about that for a moment.
Go and Rust do not have dynamic linking. Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
C# only has dynamic loading if you aren't using the AOT. That means you're JITing. I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
And pam is engineered around dynamic linking. That's sort of the whole point of how it works. Your pam stack refers to modules that get dynamically loaded, which then allows pam modules to pull in things loads of other libraries.
So: there is currently no scalable alternative to Fil-C for making pam safe.
> More and more C libraries are getting pure rust ports these days.
I don't think these rewrites will ever be mainstream. There is so much C out there, and typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
> I don't think these rewrites will ever be mainstream.
Forever is a long time. And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
Whether or not new implementations become valuable and get maintained over time is an open question. But the energy is definitely there. (Even though its not necessarily pointed in the direction of economically useful, long lived software.)
> typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
I hear a claim like: "The current C version of most tools is the only place, and only project which will ever bother solving its chosen problem properly." I'm not sure if tahts quite what you mean, but its a wildly bold claim. Even in C and C++, this seems already false. There's several good, solid, standards compliant SQL databases. And OS kernels. And compilers. And web browsers. If there's room for several C based OS kernels, do you really think there'll never be a viable OS kernel in rust, Zig or Odin? Or a web browser or compiler?
It sounds like Fil-C combines the ergonomics of C with the performance of Python.
It might have a beautiful niche for safely sandboxing legacy code. But I don’t see any compelling reason to use it for new code. Modern GC languages are much more ergonomic than C and better optimised. C#, Java, JavaScript and Go are all easier to write, they have better tooling and they will probably all perform as well or better than Fil-C in its current state.
On the topic of Java.... did you know this? The original inventor of Java is a convicted pedophile. It's so embarassing that they scrubbed his name from pretty much everything, and when you Google "creator of Java" it gives you a different person (someone who got involved later). Some useful trivia for the future. Might come in handy.
I was reading the `fontconfig` source recently (to help me understand what, exactly, is the difference between the various `fc-*` tools and their options) and some of the code scared me.
I thought, "for sure this is a use-after-free" ... but it happened to be safe because some function performed an undocumented incref, and the return value of that function was kept alive.
So this is definitely a high priority thing to port!
... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
> ... I suppose the interesting question is: if some of my dependencies are ported, and some are not, is there any way a normal compiler can call into a Fil-C-compiled library for just a few functions?
To my understanding, there's no way to do this: Fil-C is basically a managed runtime for native code, so the FFI implications are similar to those for Go or any other intrusive managed runtime. Which is to say that Fil-C could offer an FFI, but not without losing the blanket exit-instead-of-memory-unsafety guarantees it aims to offer.
Exactly right
If Fil-C requires complete recompile of ".c" code how does it deal with calls to the OS - Does it rewrap them (like Go?). I'm bit uncertain here...
It uses a sandwich; see https://fil-c.org/runtime
Thank you!!!
What is the actual catch with Fil-C? Memory safety for C based projects without a complete rewrite sounds like a very good idea, but there must be some pitfalls in this approach which this would not work in some cases.
In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
But great work on this nonetheless.
Author here! :-)
> What is the actual catch with Fil-C?
Two things:
- Not ABI compatible with Yolo-C. So, you have to recompile the whole stack. This is both a major catch, and it is both a bug and a feature. It's a bug for obvious reasons. It's a feature because it means I'll probably be the first to have a totally usable, totally memory safe desktop environment.
- In my testing, it's between 1.2x and 4x slower than Yolo-C. It uses between 2x and 3x more memory. Others have observed higher overheads in certain tests (I've heard of some things being 8x slower). How much this matters depends on your perspective. Imagine running your desktop environment on a 4x slower computer with 3x less memory. You've probably done exactly this and you probably survived the experience. So the catch is: Fil-C is for folks who want the security benefits badly enough.
It's not obvious that either of these catches are fundamental. The Fil-C development philosophy is very much to get something working first, which means downscoping. That's why I didn't include Yolo-C ABI compat as a goal at all and it's also why I haven't done as many optimizations as I should have. If you look at my GitHub you'll see >20 open issues with performance optimization ideas.
> In some other memory-safety circles (Rust, Zig, etc) I'm going to expect that this is going to be heavily scrutinized over the claims made by the author of Fil-C.
Of course this has happened.
Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C). Fil-C doesn't have an `unsafe` statement and the dependencies are all recompiled with Fil-C.
> Zig is definitely not as safe as either Rust or Fil-C, since Zig doesn't have a great story for use after free.
Have you considered making... Fil-Zig? Would it be easier to do because Zig is already fairly far ahead of C in terms of safety?
> Rust is less safe than Fil-C in practice, because Rust code uses `unsafe` a lot (>100 uses of `unsafe` in uutils and sudo-rs, for example), and Rust code ends up depending on an unsafe stack (like calling into libc, but also lots of other dependent libraries that are written in C).
To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe", it's just that PAM isn't any safer (because it's unmodified C). Also you can use Rust without libc - https://github.com/sunfishcode/mustang - unfortunately it doesn't seem to have gained much traction which I think is a shame because glibc is a curse.
Most Rust programs actually have very few C dependencies. The big exceptions are libc and OpenSSL, but I think they'll be excised eventually.
> To be fair `sudo-rs`'s usage of unsafe is all there just to interface with C code (e.g. PAM) so it isn't really sudo-rs that is "less safe"
That's exactly my point.
sudo compiled with Fil-C: uses pam compiled with Fil-C, and all of pam's dependencies are compiled with Fil-C, so the whole thing is memory safe.
sudo-rs: uses pam compiled with Yolo-C, so it's not actually safe. pam is quite big and pulls in other unsafe dependencies
[flagged]
Almost all uses of unsafe in Rust are either for FFI, or to avoid the overhead of things like ref-counting. If you use, eg, Rc RefCell everywhere you will achieve safety without issue.
> Almost all uses of unsafe in Rust are either for FFI
Which makes Rust less safe than Fil-C.
Consider that you have a dependency like libc, PAM, openssl, or the like.
In Rust: you will `unsafe` call into the unsafe versions of those libraries. Hence, even if your Rust code is safe in isolation, your program is not safe overall because you're pulling in unsafe code.
In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
> In Fil-C: compile those dependencies with Fil-C, and then the whole process is safe.
Great work on Fil-C by the way. Very technically impressive stuff!
It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this). More and more C libraries are getting pure rust ports these days. It’s increasingly rare to actually need to pull in C libraries at all. Sticking to safe rust is a bit inconvenient - but I suspect so is using fil-c. I suppose with fil-C you can use any libraries you want, you’ve just gotta recompile them. That’s cool!
Rust has a bunch of unsafe code in std, so you’re still at risk of a bug there causing safety problems. But the same is true of Fil-C.
I could definitely see myself reaching for Fil-C when running legacy code. It’d be nice if there was a way to use fil-C to protect my rust program from memory bugs in C dependencies. But I can think of dozens of ways why that would be tricky to pull off.
Also I’m curious - can Fil-C protect against threading bugs like rust can? Rust has Send & Sync traits to mark which objects can be safely shared between threads. I can’t think of a way to do that in an automated way without help from the programmer.
> It seems like the competition for Fil-C is other garbage collected languages. Rust will always outperform Fil-C, with the caveat that rust allows unsafe blocks, raw pointers and raw C FFI. But if you’re happy to ditch performance and the ability to do bare metal systems level programming, why Fil-C over Go or C#?
Say you wanted to write a tool like sudo in C# or Go. Here's what would happen: you'd end up having to rely on dependent libraries, like pam, which then relies on other things like libselinux and libaudit. And libselinux depends on pcre2. Those things would be compiled with Yolo-C, so while your sudo tool's new code written by you would be safe, the process would mostly contain unsafe code.
But if you write sudo in Fil-C, then you can rely on all of those dependencies being compiled with Fil-C. The whole process is safe.
So, the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
> If you want a safer, less convenient dialect of rust, that also exists. Just don’t use or depend on any unsafe code in your program. (There are 3rd party tools to check this).
That's impractical for the reasons above.
> Also I’m curious - can Fil-C protect against threading bugs like rust can?
Not like Rust can, but yes, it does.
Fil-C preserves memory safety under races. This means that debugging multithreaded C/C++ programs in Fil-C is a much nicer experience than in Yolo-C. If you do something wrong, you're going to get a panic - similar to how multithreading bugs in Java or C# lead to exceptions.
But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
> the reason why you'd pick Fil-C is that it's actually memory safe for real, unlike the alternatives, which just give you safety for newly written code but pull in a ton of unsafe depenendencies.
Right; that makes a lot of sense. So currently the only implementation of pam is written in Yolo-C. If you want to build a safe, secure tool which depends on pam - like sudo - then you either need to trust pam (like sudo-rs does). Or you need some way to compile pam to safe code. Fil-C is a way to do that.
Just to name them - I can think of two other solutions to this problem:
1. Sandbox pam within a wasm container. This is probably less convenient than Fil-C (although it would allow that wasm container to be linked from yolo-c, rust, go, js, etc). For pam in particular, this would also be much less safe than Fil-C, since memory bugs in pam would still be possible, and might still result in privilege escalation. Wasm only sandboxes memory at the boundary. It wouldn't protect pam itself from its own bugs like Fil-C does.
2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
I think this also reaffirms the role I see for Fil-C: as a tool to make legacy C code (like pam) safe, at the cost of performance. For new code, I'd rather pick a fully GC language if I don't care about performance, or use rust if performance matters. Again, it would be nice if I could have the best of both worlds - use fil-c to wrap and sandbox pam, but write the rest of my program in rust or C# or Go or something else that is already safe without needing to pay fil-c's cost.
> But, unlike Rust, Fil-C does not attempt to ensure race-freedom for your own logic.
It depends what you're doing, but this can be a pretty big deal. I suppose this puts Fil-C in about the same safety and performance camp as Go, Java and C#. Ie, it has some protections that rust is missing - like there aren't any unsafe blocks. And its missing some protections rust has - like better thread safety. The unique selling point is it works with existing C code, like old C libraries that we don't want to rewrite. Thanks for your work on it!
I think the whole point is to _not_ rewrite anything.
> 2. Rewrite pam itself in a safer language, like C#, Go or Rust. Then sudo-rs doesn't need to link to yolo C code. I suspect someone will do this soon.
OK, let's think about that for a moment.
Go and Rust do not have dynamic linking. Well, Rust sort of does, if you're OK with using C ABI (i.e. `unsafe`).
C# only has dynamic loading if you aren't using the AOT. That means you're JITing. I don't think JITing scales to every process on your system doing it (you get no sharing of code between processes, and if you consider that I've got >500 processes on the machine I'm using right now, you can imagine how much of a memory burden JIT-in-everything would be).
And pam is engineered around dynamic linking. That's sort of the whole point of how it works. Your pam stack refers to modules that get dynamically loaded, which then allows pam modules to pull in things loads of other libraries.
So: there is currently no scalable alternative to Fil-C for making pam safe.
> More and more C libraries are getting pure rust ports these days.
I don't think these rewrites will ever be mainstream. There is so much C out there, and typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
> I don't think these rewrites will ever be mainstream.
Forever is a long time. And there's always a huge appetite amongst smart young developers to create their own ecosystem. After all, you don't become Linus Torvalds by contributing to linux. You become linus torvalds by creating your own operating system from scratch.
Whether or not new implementations become valuable and get maintained over time is an open question. But the energy is definitely there. (Even though its not necessarily pointed in the direction of economically useful, long lived software.)
> typically ports focus on the 90% that is easy to write and neglect the remaining 10%.
I hear a claim like: "The current C version of most tools is the only place, and only project which will ever bother solving its chosen problem properly." I'm not sure if tahts quite what you mean, but its a wildly bold claim. Even in C and C++, this seems already false. There's several good, solid, standards compliant SQL databases. And OS kernels. And compilers. And web browsers. If there's room for several C based OS kernels, do you really think there'll never be a viable OS kernel in rust, Zig or Odin? Or a web browser or compiler?
> What is the actual catch with Fil-C?
It sounds like Fil-C combines the ergonomics of C with the performance of Python.
It might have a beautiful niche for safely sandboxing legacy code. But I don’t see any compelling reason to use it for new code. Modern GC languages are much more ergonomic than C and better optimised. C#, Java, JavaScript and Go are all easier to write, they have better tooling and they will probably all perform as well or better than Fil-C in its current state.
On the topic of Java.... did you know this? The original inventor of Java is a convicted pedophile. It's so embarassing that they scrubbed his name from pretty much everything, and when you Google "creator of Java" it gives you a different person (someone who got involved later). Some useful trivia for the future. Might come in handy.
This is super gross, but I don't see it is relevant... Does java's ecosystem suddenly not solve the same problems it did before?
I think it's interesting! Don't you? A fun fact of history most people don't know.