OK, I think I see what’s going on with this, in this configuration.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
I think the conclusion that can be drawn from this is that the behavior of crypt() is howling-at-the-moon insane, and it should no longer be considered for any system built in this century, at least not without extensive wrappers that correct and normalize all these edge cases.
In this particular case, it is type safe. NULL, by the C standard does not refer to an object. It means that every pointer in C is effectively an option type, which mean that a function that returns a pointer can return NULL for "no result", which, in the case of crypt() is perfectly adequate.
It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer. C++ has references in an attempt to fix this problem.
Because C tends to prioritize performance over safety, a NULL pointer dereference is considered undefined behavior and the compiler can invoke nasal demons if it makes the code go faster, but with the right options, you can have it checked and invoke what in other languages would call a panic handler, but on UNIX is the SIGSEGV signal. You can then attach a debugger or read the core file and see what went wrong.
Note: I am not talking about "char *" for strings, that one is unsafe, but NULL is not. It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.
You might consider it type safe according to C's rules, but it is not type safe according to contemporary opinion because the success type and error type are the same.
I don't think so. Even in a language with a good type system like Rust, somebody could have made this API:
/// Hash the phrase with the given setting.
fn crypt(phrase: &[u8], setting: &[u8]) -> [u8; CRYPT_OUTPUT_SIZE];
Then, in a later version, `crypt` needs to become fallible, so they need to signal errors somehow. But because they really want to preserve backward compatibility, they can't change the output type to `Result<..., ...>`. And so they end up doing the same thing that was done in libcrypt - they output some error value and hackily make sure it's not equal to the `setting` string, under the mistaken assumption that the output of `crypt` would only ever be compared to the `setting` string. And then someone creates the exact same bug.
Maybe I'm wrong, but it looks to me like the exact same incentives and means to screw things up exist in C and in Rust for this particular bug.
IMO type safety requires both language features and a mindset where you take efforts to keep things type safe. Your example has only the former.
For example, in a Rust web app, you can choose to use String for encoded URLs, unencoded URL fragments, escaped html, unescaped HTML, usernames, etc. Or you can choose to use newtypes and carefully design all APIs to only take and return the correct newtypes.
The problem is not that you can write bad stuff in rust, the problem is that this particular usage pattern can't be sanely coded in C. This is a public API that should inform users and enforce certain properties. A Results type does exactly that, you would have to go way out of your way to force invalid stuff through that type alone.
I don't think so. In both Rust and C, the solution is the same - make the function explicitly fallible, instead of trying to preserve backward compatibility. In C, the function would probably just look like this, returning zero on success and negative on failure:
int crypt2(const char *phrase, const char *setting, char *out);
You can call such a signature "insane" if you want, and that's fine, but it would have averted the bug that is the topic of discussion.
what you are suggesting is a form of type safety. There are still degrees of type safety even in C, and this bug resulted from deciding that backwards compatibility is more important than type safety, which I think we agree on.
Specifically, that the lack of tagged unions (or really, structured return types in general; you can pretend those are a tagged union if you believe hard enough) in C library interop conventions is a terrible foundation on which to build widely-used APIs.
And that just ends up indicting the C language itself. Can't send errors down a separate channel; folks might not inspect errno. Can't return NULL on error, folks might compare it blindly to another result. No tagged unions, so can't return one. Can't crash on error because there's no scoped error handling. Can't return a struct for checking .error/.result separately, because alignment and equality (and tearing and accidental clobbering and more) make that brittle as hell for widely-used APIs.
C is not a good language for library APIs. Like, really not. It has its uses/strengths, sure, but without even getting into the typical gripes people have about C, libraries--one of the things that it is used for the most--are one of the things for which it is the least well-suited.
Yep, the linked page doesn't /feel/ AI written (maybe it's using a grok or muse model whose tone I don't quite recognize yet?) but it kinda just states a lot of facts that make sense on their own but don't cohere into a whole vulnerability like the comment above does.
I think it's just Claude prompted to write in a casual, conversational, no caps style.
The surface-level hints are gone, but the attempt to make every sentence pithy, the decisions on what information to present and what to assume the audience knows, and the over-emotionalization are all still there.
and what same human would embed two linux systems (??) in a blog post just for a demo that prints "pass"/"fail" that hardly anyone will bother to interact with?
Because Claude wrote the linked page. Claude is impressively bad at communicating these days. with gpt-5.6+ approaching Claude level like 30% of time and Gemini much clearer but also the most annoying 7 year old ever, it’s a pretty lame environment especially for people who r new to everything
I wondered about this myself as the explanations and documents I have been maintaining are becoming increasingly 'slop', but a different kind of slop.
In the past I could get an explanation, where I was able to track the line of thought. Now when I read through the explanation it is like someone has taken two or three explanations and merged them into each other, like there is a combining of iterations without clarity.
example:
The tree is blue.
The tall tree has the color of blue.
the tree, while tall, is blue.
That finally gets squished into: the blue tall tree has height and is evident of blue.
Is this double speak or injected double effort of words?
Until Opus 5.0 I'd have agreed with you, but Opus 5.5 is honestly one of the best I've ever sparred with, vastly better than the rest, including Astra imho. Especially so when prompting for clear top-down, first-principles communication.
Disagree, I review a lot of Opus 5.5 code and its comments and docs are still pretty inane and Claudish. Is it better than the 4.x and 5.0 era, sure. But it's still nouning the verbs, using colons everywhere, a-not-b, "a foos the b" etc. A lot of lingual knots that need a human hand to untie.
Opus 5.5 is much better, but IME I always have to force it to review its writing (Opus 5.5) to get it to put the most important facts first (and not bury the lede). It loves to park the main point far away after intermediate clauses. It always writes “premise, so consequence.” It takes a prompt asking to review against rules in context for it to fix it.
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
I agree. The “excuse” for implementing those behaviors is just providing plausible deniability. The commit that lets crypt return “*”, followed closely by a change that lets you easily trigger that behavior sounds just like how somebody would sneakily introduce a back-door into tons of linux systems while making it look as innocent as possible.
Well, it was intentional, but not in the way I think you mean.
musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
That was a misfeature and should never have been implemented. But I bet it was an easy band-aid change to make!
But choosing "*" as the character to return is an insane choice for anyone who knows the basics of linux sysadmin, or hell anyone who's ever used a search field in any context and thus knows it is ubiquitously used as a wildcard operator.
> musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
Of all the things to mitigate...
1) crashing should be an option - it noisily indicates "there's a problem" and if it faults and dumps core, you've got a guided tour of your problem.
2) Is testing NULL as a passable option likely? I'd guess a more likely idiom is:
I think the issue is that a lot of callers were not aware that crypt() could error at all and all the issues are downstream of that. If OpenSSH knew that crypt() could error they would have handled it (possibly by inspecting errno).
I don't think that's the problem. The man page for crypt clearly says that erroring is a possibility and that it's not standardized how it errors:
> The behavior of crypt on errors isn't well standardized. Some implementations simply can't fail (except by crashing the program), others return a null pointer or a fixed string. Most implementations don't set errno, but some do. POSIX specifies returning a null pointer and setting errno, but it defines only one possible error, ENOSYS, in the case where crypt is not supported at all.
In my opinion the problem is that the behavior of crypt is underspecified - it isn't even specified how it signals failure, which is shocking.
Another disturbing decision that crypt's signature takes in the hash its output is supposed to be compared against - if it weren't for that, this half-baked "on error the output won't compare equal to the input" measure that crypt and crypt_* use wouldn't have been possible.
You can't check errno to see if there was a failure because, as the man page says, the value of errno is unspecified when crypt succeeds.
This specific manifestation of "C is terrible for shared code APIs" is rare.
But issues caused by folks forgetting to check errno, miscomparing NULL to NULL when calling multiple APIs, dereferencing external-API-returned NULL due to missed error checking, mishandling inherited errno after successful action, or--the most common--mishandling negative integer return sentinels meant to indicate errors? Those are incredibly common.
This specific library call is quite poorly designed.
Standard C api calls return an integer, which is 0 on success or nonzero on error. Ignoring it is an intentional choice. `errno` is the hack that shouldn't really exist because it's trivially easy to forget to check that and also has issues with threading (patched by making it a thread local) and more recently, coroutines (which admit no general purpose patch because they're a userspace construct).
Another reason why this is terrible is that it's unclear what the returned pointer's lifetime is. Is it until the next call to crypt? If so, is it thread-local like errno? If it's passed on to the user, is it safe to call free() on it, or is there a crypt_free function that should be called instead?
But all of this, and I mean all of it pales in comparison to returning '*' on error.
> But all of this, and I mean all of it pales in comparison to returning '*' on error.
100% agree. This is an egregious failure. But C's error-handling story would be bad even without this, and the badness of C's error handling is what motivated the (bad) decision to return *.
It'd be great if we wrote systems software in a language that supported advanced features like...
checks notes
...returning multiple values.
Hell, even in VBA you had to at least spend the keystrokes to type "on error resume next".
It's not a C thing, it's a Linux thing. The asterisk is relevant to how crypt() is used for /etc/shadow.
If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not
be able to use a unix password to log in (but the user may log in the system by other means).
No, it's a C thing. They previously returned NULL for invalid passwd lookups, but because there's no way in C to force API users to decide what to do if an error occurrs, that was causing program crashes due to NULL dereferences.
So they made the (very bad, but motivated by C's broken design/conventions) decision to return a magic string instead.
Of all the possible strings this seems like the absolute worst choice. If I were typing that string I'd pause to think "wait, what if the caller uses this with a regex?"
One has to remember the context here. The only error possible according to the SUS is ENOSYS, literally that crypt() is not actually implemented in the library. The traditional AT&T Unix crypt() was always implemented, and its encryption did not have failure modes, so there was never any error return to handle. This is why programs were written to never check the result of crypt(). It was only ever NULL on systems where one couldn't use the mechanism at all in the first place.
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
The most charitable interpretation here is that musl authors are incompetent.
There is simply no universe in which a security-conscientious reviewer looks at the code, spots * being a valid return value, remembers that * is the default value in /etc/shadow for "no password login please" and then is all like:
This is fine, you check for * and return + instead.
There are more 32-long character sequences than atoms in the universe. Pick one, any of them instead.
Returning a string on error honestly is defensible (especially in a language like C where the type system doesn’t have exceptions or result types), the question is why the string was “*” instead of like “crypt ERROR: input password cannot be longer than 256 characters” lol
"*" is what's been historically used to represent an invalid/no value in passwd/shadow, so it makes some sense.
Of course given the chance that one might do a direct string comparison between crypt's return and the passwd/shadow file's hash field it was still a not well thought out choice.
In-band signaling[1] strikes again. * should not have been used as special value, neither for "no password" in /etc/passwd nor "error while hashing the password" in crypt. I'm afraid we'll find many more of these abuses, specially in string manipulations. The null was ok, an exception (or equivalent) better, and Result<T> (or equivalent) best.
Exceptions are almost never the right choice. They create extremely complicated control flow extremely quickly, and then you have to test all the different control flows. Java software is an example of how this goes sideways fast. There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor), and objects are default initialized, so all fields (even static ones that are assigned in the constructor) can be null.
So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:
There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.
I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.
I'd argue that's the fault of having default nullability, not the exceptions. You would have the same problem if every attribute access returned a Result<T> too.
And I'm not a fan of breaking control flow, but I still find it preferable to nulls. A null may travel far from the place where it was generated before blowing up, making troubleshooting harder, or it might come from a function you called for its side effect but forgot to check the return value. I gotta say I'm also a careful enthusiast of checked exceptions, it's great knowing which ways a function can fail.
But I think the consensus is growing that Result<T> (plus warnings for unused return values) is simply the superior choice.
As long as an error condition on the return value causes a crash on use, then it's mostly the same as result. (The main remaining difference being that you do an implicit `.unwrap()` in C, and in things like rust that's an explicit code smell.)
> There's no way to create a class that prevents callers from reading the object before the constructor runs (in multi-threaded software, new first hands a reference to the caller, which can put it on the heap, then it calls the constructor)
Which is why the pattern of double-checked locking doesn't work.
> There's no way to create a class that prevents callers from reading the object before the constructor runs
That can only happen if you leak the reference (given we are talking about java language level semantics and not byte code), which I give you is unfortunately not that hard to do.
As for exceptions, I think they would pair beautifully with software transactional memory, which would work well with effects. In that case (and no side effects) it would not matter what code path we took, we could just go back to a known safe point.
Because ergonomy-wise I still think exceptions win (even though I do like Maybe types, and Rust's ? macro is plenty good):
- they let one focus on the actual happy path code
- no error is ever swallowed, unless explicitly written so (looking at you go with useless if errs that fail to properly handle the error), they are always handled somewhere
- they automatically "unwrap" the Ok result, and automatically bubble up the Err side.
- bubbling up is especially useful because most errors can't be handled at their source (e.g. IO error is meaningless, how you handle it depends on what you want to do in the first place, which is done several layers higher)
- they automatically attach a stackframe/source, while with result types you may get a list of Ok Ok Ok Err and have no idea how that Err got there.
PSA: you can typically declare different types of string so that your type system objects to mixing data of different types even though deep down we know they're both strings.
If you mean type aliases, that's a good idea within a single codebase, but the distinction between different types of strings doesn't survive serialization so it wouldn't have prevented this error. What they need is a way to signal errors out-of-band, like Result<T> or exceptions.
In python, aliases are separate from typing.NewType, but you're right: it's only enforced within a codebase and then only if the dev cares to run the typechecker.
So yeah, I guess it either needs to happen at runtime, or we just do signaling and user data on separate file descriptors.
This is presuming that the type system is nomitive rather than structural and there is not serialisation/deserialisation or type refinement occurring. A structural difference is much more preferable.
This is a side-effect of Unix-y "it's all plain text" design that leans extremely heavily upon in-band signalling (among other design decisions).
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
A few years back I un-bashed my life as much as possible in favor of nushell (though it's not the only shell to encourage structure). I feel like it has really helped me notice a lot of this-needs-more-structure type problems outside of my shell that I otherwise wouldn't have.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
At Amazon we had RecordStream which implemented most of the standard Unix tools on structured streams. It was pretty easy to write filters, aggregators, and formatted output in something not much more complicated than sed/awk/cut/join/sort pipelines. There was a convenient structured log format for services that was often used as input.
That looks really cool. I've been heading the opposite direction: posix shell for everything.
I have a bunch of management scripts, etc, that need to run on everything from a potato to a T-1000, and that's the only language that's always available. (OpenBSD sh, FreeBSD sh, busybox, bash, dash, zsh all can cope with it).
I'd love something that's at least JSON-typed (or equivalent), but that can be bootstrapped up on least-common-denominator posix.
Has anyone built something similar? (Yes, I'm being picky -- bun / ts is already a pretty low bar for cross platform support, but these scripts do stuff like install node, so I'm not totally crazy for asking.)
The issue isn't the config, but the fact that crypt() doesn't have any way to signal failure. In better designed languages you would have exceptions, result types, or something. Even in C you have "return integer indicating success or error" which would've prevented the confusion.
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
>Upon successful completion, crypt() shall return a pointer to the encoded string. The first two characters of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
OP's sshhole link specifically notes that it _should_ return NULL, but some MUSL devs decided they wanted it to return trash instead, and the trash they returned instead of NULL caused this.
All because some devs were too lazy to check the result and another dev decided to be clever and "fix" NULL errors from crypt by sending back 'unmatchable' trash banking on devs not checking NULL.
edit: apparently crypt implementations have been trash ignoring the NULL return for a long time. this isn't only on MUSL, but it's still because people ignored and tried to work around the existing out-of-band error mechanism.
But all those sentinels were added for the reason that, when your code is a public API intended to be widely used by who-knows-who, it needs to be really difficult to misuse it. NULL-plus-errno is easy to misuse. Yes, it's a dangerous mistake and your own fault if you do it. But people have been making that mistake for years and don't show any signs of stopping.
> returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
> OP's sshhole link specifically notes that it _should_ return NULL
I meant there's no mechanism besides returning NULL that would make it hard to ignore the issue (exceptions need to be caught, results need to be unwrapped and their failure handled, an integer return value makes it clearer that there's something there that you need to check). I also missed errno codes when glancing at the manpage earlier which should illustrate the problem.
And yes, whoever first had the brilliant idea to start returning garbage never should've done that, I'm not disputing that.
People don't RTFM, either on HN or when implementing "standards compliant" stuff like MUSL.
No error handling scheme can fix that problem. For instance, if a Rust API says I should return a Result<String, ENOSYS>, I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS).
Sure, but if you didn't do that, and returned the tagged union (in Rust or whatever language), your users would necessarily be forced to handle it in order to process the result.
The only reason this library did the insane thing of returning "*" is because it was extremely easy to cause crashes or false-positives (NULL == NULL) by returning NULL+errno. The reason that was easy is because C lacks a means of making error handling mandatory (exceptions, tuple returns, tagged unions, whatever).
If you return a tagged union, users might mis-handle it, it's true (https://blog.cloudflare.com/18-november-2025-outage/). But they have to handle it, which is a requirement that C APIs are incapable of providing.
You have to handle NULL in C, or your program crashes (though I suppose a misguided optimization of strcmp could do "a == b || really_strcmp(a, b)", which gets us back to your bug).
Also, in things like rust you can just do result.unwrap() if you want. In general, language constructs don't really save you from bad programmers.
> I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS)
You can do that but it's going to feel bad to write that and it's going to stand out like a sore thumb to anyone who reads it. But "errno" is just ambient state that's completely invisible unless you happen to have the manpage fresh in your memory.
Yup, that's the way to explain it in order to blame musl.
Here's how to explain it in order to blame openssh:
openssh processes passwords and doesn't want to reveal through timing whether a user has a valid password (or exists at all). So far so good.
What they do if they hit a user with no valid password (i.e. "*") is that they look for another user with a valid password, to use the seed from. A bit weird already, but OK from a perspective of avoiding implementation specifics on seeds and hash algorithms. They know at this point that the result must be to deny access even if the hashed value matches, because that hash is for another user.
So they put the password received over ssh into crypt() with this other user's hashed value to get a seed. And then accept it if it matches, apparently?
Homework: what happens if the password actually matches another user? Can I log in as root if openssh randomly picks my user for the fake salt, and I type my own password correctly?
Personally I think there was nothing wrong with returning NULL. "But if people don't check for it it crashes their programs!" Good! Let it crash loudly! Then they notice and fix the issue.
Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.
> Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.
I wonder if somebody else would have thought the exact same thing in a different (non-/etc/*) context, and then you get the same effect in a more obscure manner.
Some badly designed password hashing algorithms (cough bcrypt cough) have maximum password lengths, ignoring everything after X characters. So it could happen that I set my password to a 100-character string, it gets saved as "==DANGEROUS_CRYPT_ERROR_length==", and then anybody can log in to my account by entering an unrelated long string that returns the same "error".
If a function promises a string result, it should only provide a string if the function succeeded. The null was much better than magic strings.
I know, it's bad. I agree that it should've stayed NULL. But if you really really really think you have an acceptable reason to do something like this (and you probably don't), the least you can do is use a non-trivial token.
If I was really desperate and could only return a valid string, I would probably use a message and add a random component to avoid ever matching two errors. Like SQL, where null != null.
Returning “*” instead of NULL to signal failure is the real footgun. Don’t blame the format.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
"panic" is a good thing. There's always some error condition that your code cannot handle. At some point, the least bad thing to do is crash.
For instance, what if your error handling code throws?
One spectacular example of panic saving the day was the Apollo 11 1202 error. It was basically a watchdog timer that fired if the landing computer was starved of CPU time. A misconfiguration caused the first moon landing to hit CPU overload. When that happened, the computer displayed 1202 and rebooted, then ran things in priority order. Apollo 11 landed in the middle of a tight 'panic, reboot, adjust thrusters' loop. Thanks to core memory, most of the heap survived the reboot, so the reboot step was fast enough for it to mostly land without human interaction. The pilot had to make ~1 adjustment to get it to actually hit the landing site, but the panic-ridden computer handled the rest of the descent.
sshole-type APIs < panicking APIs < required error handling.
Panic is fine, but fundamentally a whole-program crash is undesirable in a really wide range of use-cases. I don't want other SSH connections to crash (or the server to stop accepting valid logins until someone restarts sshd) if one person tries to log in with a credential that triggers a panicking bug in crypt(). Same for crashes in request handlers and so on.
The language/runtime itself doesn't (and probably cannot, at least if it's a general purpose language) distinguish between local and global error conditions, so aborts end up being either unnecessarily damaging, or languages like Go end up normalizing panic "localization"/handling in dangerous ways. Isolation using threads isn't robust enough. Isolation using processes is far too fiddly, slow, and opaque.
The better solution is required error handling. If user code must engage with the possibility of an error state in order to access a result, that's a huge step forward. They can do the error handling wrong, but if you make most people have to do it at all most of the problems will go away.
There are other good solutions too, like Erlang's process boundaries, but those tend to have drawbacks that make them inapplicable to a lot of programming areas. Tagged unions, on the other hand, are only inapplicable in C library code because of language defects and terrible community conventions around ABIs.
"I forgot to check for an error" should not be implicated in bugs in 2026; we have had a large menu of possible alternatives (checked exceptions, unchecked exceptions, tagged unions, etc.) for decades. But we're still over here writing C (and Go), languages pathologically incapable of providing this ability.
SSH specifically has separate processes per connection the last time I checked, so a panic does the right thing: The attacker's connection drops uncleanly.
This is also why you can upgrade SSH over SSH without having the connection drop mid-stream.
Again: You cannot avoid "I forgot to check for an error". What happens if you get a sigpipe on stdout when printing the "I crashed" line? If Linux overcommits memory? If you turn off overcommit, port all software to check for + cleanly handle NULL from malloc (a substantial effort!), but then get a machine check exception from a double bit ECC error during an infallible read?
Sort of. It does some of the validation/auth code in the sshd parent daemon.
It has to: if it did 100% of the post-accept(2) work in a subprocess it would be pretty easy to remotely forkbomb/PID-exhaust a system by spamming invalid logins.
> You cannot avoid "I forgot to check for an error"
Yes you can. Checked exceptions and tagged unions both force calling code to be written with error-awareness.
In SSH's case, "I forgot to check" might lead to crashes (which might knock over sshd so new connections are impossible), or it might lead to false-positives via comparing NULL == NULL.
I'd much rather that boneheaded API users have to write code like
Random parts of it will be interpreted by the shell, others by cURL, or the filesystem, or the remote server. They each have their own encoding rules, nested. I count ten distinct separators and nine ways of escaping/tagging. It's one command. No, just no.
We have seriously overused strings and we're now paying the price.
At this point I’m resigning myself to think that we’ve just gotta wait for some people to die. It feels very dramatic and glib (no pun intended lol) and whatever to say, but at this point it feels like there’s a certain hostility in some parts to changing anything from the platonic ideals of however computing happened to work in 2004. “Oh we don’t need to mess with it!”
How did you arrive at that conclusion? If anything, this vulnerability happened because someone decided to change the behaviour of crypt() without fully understanding the consequences. This specific vulnerability did not exist in 2004.
I agree with him. I only managed to get asterisks enabled by default in sudo because sudo-rs isn't owned by people who have decided that the 70s was the perfect decade and nothing should ever change. And look at the insane pushback when it did!
I can't even get an obvious UX improvement through PAM because they like it broken.
I think this is a bit unfair. The way I see it, a lot of people are saying that changing breaks lots of old stuff, and risks complete fragmentation. Those are real costs to change. It might still be worth it, but let's not pretend that everyone is resisting change just because it's change.
I'd say another UNIX-y thing - "C is good enough" - is part of the problem. C makes programming painful which results in all sorts of hacks.
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
Pascal's record typed files have the exact same problems regarding endianness and word size as directly writing structs to files in C, so I don't see how they're any better. They're also a complete deadend and unusable for pretty much any file format from the last 40 years+, since you can't even handle a header or magic bytes at the start.
That would've happened to fix it, because "error_while_hashing" and "no_password" happen to not be valid password hashes. The right way would've been to have different syntax for hashes and non-hashes; you could've made it so that a hash starts with a '#' and if it doesn't start with a '#', the rest of the string should be ignored.
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
Well neither * nor ! (the two characters I've seen to indicate "no password") will appear in a valid hash. The problem was using a character that could also be returned as an error indicator, IIUC.
Yeah but it's not forwards compatible with future hashes where '*' is suddenly a legal hash, which is what happened here. You need a grammar which lets you tell the difference at the parsing stage between "this is a password hash" and "this account has no password". That's not the same as letting the value '*' flow through the system as normal and hope it doesn't happen to match the hash.
Right but not in a "rigorous" way. They kept the format of the file the same: the string between the colon characters is a hash. They exploited the fact that an asterisk happens to never appear in a hash, and just let the system believe the asterisk was supposed to be a hash to be matched again.
In a sense, they should've changed the parsing logic to return a struct with a field 'password_hash: Option<String>', and defined the grammar of the field to be: "a hash string, or '*' for no login". The bug is, in part, caused by the haphazard way they don't change any formal grammar or contract and just relied on the fact that '' happens to* never be a valid hash.
It's the difference between "happens to work" and "defined to work by construction". The former approach breaks once a hash function is invented where '*' is a legal hash (such as this new musl crypt() hash function where all strings longer than 256 bytes hash to '*'). The latter keeps working regardless because the asterisk is treated as "no password" already at the parser stage.
The vulnerable path in musl is only a little bit vulnerable, because "*" is not actually a sha256 encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
> (hey, this workaround is wrong if the salt is "x"!)
When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
> Turns out that sshd detects that the password hash is too short and does not forward the salt!
I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
> When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
I think you're right. That's what I get for doing a very very quick read.
> I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
I don't care what the intent of the check is. If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in! Don't deliberately pass nonsense into the live check and rely on the nonsense check failing to prevent the fake check from accidentally working. And don't have a code path that thinks it's checking the real password but passes in the wrong second parameter to crypt().
Or, better yet, have an entirely separate function fake_check_password() that does the fake check, returns void, and cannot possibly be confused with the actual password check code.
This is why I don’t fully buy the argument that software that’s been around for a long time is more secure and reliable because it’s been battle-tested. There’s been a lot of time for cruft to accumulate until no-one can ever fix it because everyone is expecting it to function in the broken manner it’s always functioned. And some of the more obscure configuration options get a lot less scrutiny.
My general rule of thumb is: "no magic strings". This is much easier to implement in higher-level languages and when writing application/web code than system programming.
Don't return empty string/asterisk/anything else. Instead, return a failure value.
When absolutely needed, choose the magic string with more caution:
One string showing the SHA-256 of a non-existing hash? It's a good idea for it to be a valid hash. Now we have a new risk: if it's broken once, it's broken for all. So it's a good idea for it to be a random one for each machine and/or account. Even this might have problems I can't think of, but it seems to be as secure as an account with a real password.
If this is not a good decision for some reason, choose something unmistakable: "--ACCOUNT-HAS-NO-PASSWORD--". This is much less likely to be mistakenly used in a different context, is much easier to look up in a code base than "*", and is self-documenting.
Above all, I’m a little sad that the musl developers decided to make a lot of “well it’s convenient” changes to core crypto functions — at least without the level of auditing that would have caught this! It’s tarnished their reputation in my eyes a little.
I think everyone is over fixated on AI finding complex bugs beyond a typical person's understanding either to locate or exploit.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
Finding more and more that these complex bugs people gloss over like you are now are being used in exploit chains and the end result is still my clients getting hacked
This one might be hard to find using the author’s approach of throwing everything at various builds of ssh.
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
Honestly the ideas around musl are interesting, especially being able to statically link, but given the sort of nonsense around how they handled pthreads, and generally other trade offs, it's hard to imagine wanting to use it over glibc (which everyone shits on) if you don't absolutely need static linking.
In many ways each individual decision makes some sense;
wanting to stop loads of programs being affected by a (remotely triggerable?) crash
wanting to avoid time based attacks
wanting to be able to set accounts to not have a password
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
I would be more concerned about the lack of explicit return types, one of the arguments for strongly typed languages is that you should not have ambiguous data and yet…
Granted I mostly work with high level programming languages so I may be missing something.
Yes this is C, C is not strongly typed at all and the type system is very primitive. NULL ptr is the return value you should probably use if something goes wrong. But you can’t, say, return a sum type with the result and an error. It’s not uncommon at all to use magic sentinel values in C to indicate error conditions.
Any use of sum types would have prevented this immediately.
The whole thing exists because APIs that return a NULL value that "nobody checks", plus a string that you shouldn't use if you actually did check the null value.
In languages with sum types, this cannot happen.
Either you get your hash, or no result at all.
You couldn't get eg NULL and *.
This argument is weak. C, and specifically libc, has errno. The error code is there. Not in band.
But there has to be something in the payload field when it is not applicable. A null is perfectly fine.
The argument for changing that to "*", if I understand the above linked discussion correctly, was that some users of this api threw away errors and a dummy value would be harder to ignore. Maybe one could argue that these behaviours are less common in Rust or Go, but not because of a technical difference.
The world is not perfect and sometimes you really need to work around bad users. Even in Rust or Go. I am not convinced this was such a case. If you are convinced it was it would still have been better to select a less trivial dummy value (such as "*:ERROR:*").
> But there has to be something in the payload field when it is not applicable
Not in languages with ergonomic sum types. A `Result` or `Either` will have no payload field on the error type. The compiler won't let you dereference it.
It is extremely unidiomatic to use tagged unions in C for error handling. It's definitely possible, but awkward, and the compiler doesn't check that you checked the tag.
Yet the authors of crypt did not use them. The C language is not designed in a way to encourage people to write code that correctly uses tagged unions (or nullable pointers for that matter).
Nowhere. It's the same argument as upthread. A language that forces downstream consumers of an API to correctly deal with all possible outcomes would have prevented this.
Goalposts are fine. If crypt did return union type, C still allows cowboy programmers to straight extract the char* without any check whether it's valid. So we'd end up in exactly the same situation.
Because in fact all the changes were about changing the function signatures. The function signature is enforced by the linker. The semantic of strings that go through isn't.
Returning 2 values is possible in C (via pointers input parameters). The main issue here is that libc is a strong API contract based on C function signatures, that nobody can change, only augment with new symbols. Such strong API compatibility constraints happen in any programming language, and workaround to fit new behaviours in old APIs happens everywhere.
I don't think nh2 is saying that this is how the "issue" should've been solved here, but that it's a fatal flaw of C itself (yes I know, unions, but no one uses them).
But also, its not like there were many alternative languages when the POSIX standards were designed so it's fair that it works like this.
means that most ways of returning an error from a function are not available in C. Because of this, many C APIs, including `crypt` in this case, resort to in-band error signalling by using a sentinel value (like NULL, a negative integer, "*" etc.).
Most other languages lack one or two of the above, but very few lack all three, and so most have developed a different culture around error reporting (Result in Rust, multiple return values in Go, exceptions in Java, Python etc.).
One of the small but important significances of Rust is that it very often gets to have the same machine code as a correct "in-band error signalling" strategy from C but with source code that the maintainers can actually understand.
And really often when your "in-band error signalling" in Rust is more expensive you realise (perhaps to your horror) that your C wouldn't be correct, so you just dodged a bullet. For example Option<OwnedFd> is, like OwnedFd, a 32-bit value, Unix has always reserved -1 to mean invalid, so Rust can treat that as a niche. But the Windows equivalent Option<OwnedHandle> can't do this because, well, cue Raymond Chen: https://devblogs.microsoft.com/oldnewthing/20040302-00/?p=40...
> One of the small but important significances of Rust is that it very often gets to have the same machine code as a correct "in-band error signalling" strategy from C but with source code that the maintainers can actually understand.
But aside from NonZero- and NonNull-based layout optimizations, I don't think Rust is doing anything special to obtain that machine code - it's just relying on LLVM to optimize things.
not in a way that forces the user to actually handle them properly. C is a weakly typed language and as such is pretty badly suited to systems programming.
What is a weakly typed language to you? To me, implicit type conversions are a big part of the definition, and C is flush with those (void* converts to any pointer, all kinds of number conversions, arrays decay to pointers, functions convert to pointers, lvalue conversion strips const/volatile etc.).
However, I also think "weakly typed" is not a well-defined concept, everyone seems to apply it differently. In particular, it seems to mostly mean "playing it loose with types in a way that I don't like".
Yes, it is not well-defined. But most conversions, function calls, and accesses are strongly typed in C, so I would not call it weakly typed.
Implicit conversion could be seen as weak typing, but it is mostly limited to specific arithmetic operations in C. I would see implicit conversions as problematic if they allowed operations that make no sense. Decay of arrays and function to pointer conversion also do not seem problematic to me for this reason.
Void pointers are also weakly typed, but they are specifically designed for this. So if you do not want this behavior, one would simply not use them. It is a bit like using "unsafe" in Rust.
There are certainly stronger typed languages, but on the full range C is more on the strongly than weakly typed side in my opinion. A language that allows arbitrary implicit conversions between pointers, would be weakly typed, or which allows calling functions with wrongly typed arguments (which was true in the past for C a long time ago)
C is definitely not the most weakly typed (e.g. Javascript/PHP are both weaker), but IMO C is weaker than python in that python will give you a runtime type error if you do something invalid, while in C you cast something and just get UB. Most higher level strongly typed languages are able to make their casts checked because they have generics to remove the need for 99% of them, but C is missing those features and thus has to rely much more heavily on unchecked and unsafe runtime casts.
C unions are sum types if you just "draw the rest of the fucking owl". C has sum types the way x86 assembly has function calls. You see, there are "call" and "ret" instructions. Just figure out calling convention and the stack yourself, it's easy. And yes I can stick the tag next to the union myself in C and write a bunch of functions that hopefully deal with it correctly. I can even write some macros that look like pattern matching and maybe convince myself that it's a language feature, but it just isn't.
And it's very evident why the difference matters. People gravitate to what's readily available and ergonomic. That's why every C API under the sun (most definitely including libc itself) does janky stuff like returning -1 or "*" on error instead of building their own bespoke Result-style abstraction. A programming language is not just the things that are in some way possible within it, but more importantly the things that are easy in it.
While I agree that this should come out of the box, the "rest of the fucking owl" is a tiny macro or function, so not exactly something difficult to do complex as "drawing an owl" would imply.
Sure, when I last wrote a sizeable chunk of C, I wrote those macros and with a bit of help from new-ish compiler warnings, I got a reasonable error handling system in C. But whenever I interacted with anyone else's code, they apparently didn't get the memo that this is possible and desirable.
A language should encourage writing robust/secure code, not just make it possible.
While I see the point, I am not entirely sure I agree. I kind of like that C is not an opinionated language which tells you exactly how to write things. This comes at the cost that many people will do things in a bad way, and this is difficult to fix, but still probably easier than getting them to move to a different language.
In any case, I do not think the issue here is caused by not having a pre-defined result type. Checking for null would have been fine. The issue was that to accommodate old and broken code a compromise was made. But as long such compromises are done even where fixing such old code would be quite easy, it is clear that changing to another language is completely unrealistic. So what are we even talking about? The real issue is that nobody wants to spend money or effort maintaining the old code. No language can fix this.
There are different amounts of effort you can pour in to this depending on your requirements, and we can argue about those, but to me the evidence is clear: It's too much of a hassle for library authors to actually go and do it. Everyone just uses sentinel values instead.
Rust and go still make it pretty easy to not check results if you don’t care about the success value (lots of code just does `let _ = foo()`), but if you do want the value (like with a hashing function), Rust makes it really hard to not check the error too. There’s no syntax to say “assume this worked and give me the success value, including whatever garbage is there if it’s not set”:
- result.unwrap() will panic if it failed
- result.ok() throws away the error but gives you a None value instead (still can’t use it as if it’s a success)
- matching on the Ok variant and doing nothing on the Err variant, you only get to use the success value inside the Ok arm
Etc etc.
The problem is when the compiler says “NULL is a valid instance of T”, which go lets you do, and rust doesn’t (without using the unsafe keyword and intentionally casting a null pointer, which you don’t do by accident.) Go lets you write code like `var hash string; hash, _ = hashfn()` and it compiles and now you have null instead of a valid hash.
I understand that other languages could help avoiding the semi-predicate problem. That being said, in C there are better ways around semi-predicates, as inelegant as they may be, is possble, is not an inherent problem of the language, but may a bad habit/idiom of it.
I am sorry, a little? I am deeply disappointed that libc decided to do the same just with another sequence.
And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.
You must be new here. While I applauded the idea of musl when it first came out (coincidentally, glibc wasn't having the best time in its development then), the developer's arguments why <some behavior drastically different from every libc under the sun> was akchyually correct, which was why you had to fix a program that hadn't been broken for 30 years and now suddenly was, made me put him under the "clowns" category and never look at musl seriously again.
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
I’m also reminded of the clowns pushing “it’s smaller!” container images based on social media vibes rather than analysis where, yes, not using Debian+glibc saved tens of megabytes on the base OS but did so at the expense of measurable performance reductions & compatibility issues like the one you mentioned, and literally every single one of those dudes I worked with was shipping containers pushing a gigabyte so the size reduction had no measurable impact on deployments.
Let's be real here, the main culprit is C. People love to say "but you can still get logic errors with Rust", and no shit of course you can. But it's much less likely. You wouldn't have gotten this bug.
A little?!? I’m sorry, but musl is a joke, this is yet another proof of that fact, and I have no idea why anyone uses it in production, they have only themselves to blame after the many warning signs.
Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem. Silent compatibility hacks are the worst, they guarantee problems never get fixed and the stack of hacks never gets smaller.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
For security it's better, but in many cases if you're not writing security-oriented software it can be worse from a user perspective. I remember the first job I had over 20 years ago was working on VoD streaming and one of the "tricks" to improve stability was mmap'ing the 0 page so that NULL dereferences wouldn't immediately segfault. Ugly as hell but when you were working with all sorts of more or less well coded media codecs and security wasn't a concern at all (worst case scenario the user would stream movies without paying) it made some amount of sense.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
> because having a critical piece of live media production reboot mid
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
> if you're not writing security-oriented software
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
It depends on the time. Twenty or thirty years most software was not security-oriented. Nowadays, everything is online and therefore everything needs to be vigilant about security.
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem.
Agreed. I feel like this attitude has been holding Python back, too. It's one thing to criticize "move fast and break things", but it's also not good to ignore already existing breakage just because the pieces still fit together neatly, and it's not good to refuse to move at all.
I'd still rather have the software crash and alert a human that to continue in a faulty state. The way to add robustness in this case is to add backup systems, not "ON ERROR RESUME NEXT".
Now everything installs to a user local folder, runs as the user, there are no UAC prompts, and malware can just freely tamper with other applications, act as the user, and harvest all data the user has access to.
Hey, but at least it can’t touch the System32 folder.
Did anyone get it to work? I tried on the official alpine docker image, enabled password login for root. The shadow entry has ! as password by default, this didn't work with a > 256 character password (i also did passwd -l which also does !). I manually changed it to * and it still didn't work.
Ahh! Wasn't working on 3.16, which does indeed have an updated package.
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
In Alpine, these would all be the case for an out of the box install with openssh added unless you specifically install openssh-server-pam and configure sshd_config to use PAM (Alpine is already musl-c based and doesn't default to PAM).
Doesn't it also need a user to be added with a password entry?
I only ever see Alpine Linux used for containers which don't need accounts with interactive user logins, which is not to say it doesn't exist, just that it doesn't seem like Alpine comes with all preconditions met out of the box
Isn't it the opposite, accounts which have no password and hence had * instead of a hash?
> accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
But yeah, Alpine for persistant installs isn't unheard of - just not as popular as the container use case. They offer official VM, Xen, Pi, and bare metal images users deploy as well. I actually had a fleet of VMs to update!
Edit: The Alpine notice made it clearer to me that at least one other account has to have a password, it's just the target account that doesn't. So if you truly have no users with passwords in your container deployment then those passwordless users aren't vulnerable to this bug, but any of the image types which have a user will likely have root exposed.
> The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash.
No, didn't work for me open OpenWRT x64 running in a container but that is probably 'cause I'm using dropbear instead of openssh-server. That's what OpenWRT uses by default so the impact there will be minimal assuming that most installations don't change to openssh-server - it isn't installed by default.
I can't reproduce it either. I used alpine:3.24.1 and openssh is openssh-server-10.3_p1-r2
I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
> some default container images apparently have a default alpine user with that hash
Do you have an example? Looking at Alpine's advisory, I can only see that their cloud images (i.e. what you use to run a VM, not a container) have a default alpine user, no container images. And even with those, cloud images with cloud-init are unaffected with default config due to `PasswordAuthentication no` being inserted by cloud-init into sshd_config. So that leaves... Cloud images that use Tiny Cloud for provisioning. That's an instance bootstrapper authored by Alpine from the looks of it so I'm not sure how many people would prefer that over cloud-init (which a lot of VM/bare metal management software may already make easy to use) or just a non-cloud image.
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
In some contexts (e.g. embedded) you'd be surprised at how reluctant some people are to upgrade when they can run through a dozen of hoops for workarounds such as changing hash instead...
In my last org I was in charge of internal testing and qualifying software releases for network equipment, and designing test harnesses and tests for these. Significant install base. All the major vendors. Many years. Basically doing the vendor QA for them, because we couldn't afford things to fail. As you do - I wish.
This was not security testing as such - mostly latency, performance and hardcore network "can we break it" and "does it comply" stuff, but also a lot of cable and module / line card pulls, pluggable and fiber breaks, random reboots and pre-empting a lot of possible operator inventiveness. At one point we considered SEU bit flip testing on rented cyclotron time.
And always we battled against the top brass urging us to speed up the process and don't bother with "really basic" testing, because we observed software quality going down over the years as the push for release and time to market stared to dominate.
So we always started with sanity testing - and that included oversized password and CLI input strings. Always. And we caught things more than once, but never like this.
Offtopic: this is a very cool way to present commits and code excerpts. Love classic Windows styling.
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
To be honest, I love this sort of vibe-coding. Explains it really well, in an interesting way. If this is vibe-coding, then the AI has done a great job.
It was fun to see this type of style around the release of Opus 4.6, but its really tired now. I've seen countless landing pages like this...its basically ritual
There's nothing great about this site. It should give you a succinct summary of the vulnerability right up front, so you can determine if it applies to you. The history lesson is not relevant to the vast majority of visitors.
The terrifying decision for the API to return "*" on failure is a second-order consequence of memory unsafety, because it was a half-baked attempt to mitigate a common source of UB that people were encountering in the wild.
That was a contributing factor, but there are plenty of other instances of C APIs that make use of in-band signaling like this. That the API was changed from returning one in-band signal to another is because the original value was causing programs to crash unpredictably due to UB.
I expected let another AI slope website and instead that was a real human writing about what they found. Sure they used LLMs to find this, but they used their real fingers to write this. See, this isn’t that difficult, and it’s a lot more enjoyable to read.
A common complaint I've seen (and I experience on occasion) is AI agents being a bit silly in their overprotective coding style. They nit pick and catch every single little error. This can be frustrating since a lot of the time they are trying to handle edge cases that are so unlikely to happen, yet to cover them requires significant additional code.
This might be the beginning of the reason we shouldn't be reading the code. It's like in chess engines where the players complain about how weird, alien and unintuitive the moves are. But 100 steps down the line the engine finds some edge case exploit that gives it enough advantage to win.
So we may be crossing a line, right now, where the non-intuitive alien code being generated by agents is seeing edge cases down the line that might allow exploits. And if we "correct" the agent, we may be leaving ourselves open to exploits we don't see and have difficulty understanding.
The answer is not to reduce our understanding. That is the worst possible conclusion we could draw from what we're seeing. The answer is to increase it.
Slop, no question. There's at least 10 Claudisms on the page, and I'm not that good at smelling them. Some of the sentences might have been human-written.
I think at some point I'm going to get banned from HN for repeatedly saying this, but here we go:
This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.
It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.
I had this classmate that had their ChatGPT instructions to respond in this same laidback, lowercase, gen-z style. i recognized the output as a lot of llm writing almost immediately
Maybe that's on you though. I enjoyed it, though the discrepancy between the `*sshole` name and the `sshhole.com` domain made me twitch. How is anyone[0] obsessive enough to find this and write it up and yet not obsessive enough to make the amount of `h` consistent?
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
It has very low information density. This would read 10x if it were just a few paragraphs of text. Instead we need to render an animation every few sentences. Not a fan.
I do actually. My guess is that this was not just one-prompted, but someone took the time to refine it based on their tastes and expectations until it matched something they would actually like.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
It's amazing how similar this exploit payload is to the 20+ year-old Quake 3 vulnerabilities found by Luigi Auriemma (aluigi), which also used the exact same 256-to-1024 character range and nothing more than a long string of AAAA.... Here you're feeding >256 A's up to a 1024-byte limit to trip musl's defensive length cap, whereas back in the mid-2000s, Luigi was famous for publishing one-line .cfg scripts where the entire exploit was literally "say aaaaa..." repeated a few hundred times overflowing server's memory.¹ ²
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
Yes, correct for OpenWrt. OpenWrt defaults to dropbear for both server and client. However, more advanced users may optionally choose to replace it with OpenSSH, or install OpenSSH side-by-side, to gain additional features of OpenSSH.
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
sftp hn@ping.ohftp.org
There's no firewall and no password. Hack away HN.
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
Many, many Docker containers rely on "docker exec" or the kubectl equivalent to gain CLI access when needed. To have sshd in an Alpine Docker container, you're generally need to be running init and a bunch of demons, not just a single standalone service.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
Wow. I wonder which of the common fuzzers would have found this a decade ago. If literally any password over 256 chars works, that means nobody has ever fuzzed openssh on musl linux
To be fair, I have less than zero sympathy for anyone who runs SSH on any platform without `PasswordAuthentication=no` in 2026.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
> Its not asking for trouble, it is there will be trouble, not if but when.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
This is a sampling error - we're only hearing about the public disclosures.
(almost) No company actually has secret source code, only source code not publicly viewable. And the LLMs crawling those don't belong to people who make public disclosures.
> > Because a 20 character random password is comparable in brute force complexity to an SSH key.
> You can't be serious with your comment. \n\n No its not.
If you differ in opinion, maybe provide evidence? Especially for a claim like long randomly generated value 1 being absurdly ("can't be serious"-level) weak as compared to long randomly generated value 2
If you mean it's log(62^20,2)≈119 bits of entropy instead of 128+ and so it's theoretically weaker when we build a Dyson sphere around the sun to brute force that key (after somehow obtaining the shadow file, since an online attack would be way too slow), then sure
How is password auth weaker than key auth? Bugs in auth code can happen in both pathways. And apart from stupidly short passwords both should be equally impossible to brute-force.
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
Password gets sent to the server. Private key does not get sent to the server, and authentication protocol can be forwarded safely, enabling things like ssh-agent, FIDO SSH keys. This is solvable with passwords (see: augmented PAKE), but `PasswordAuthentication` isn’t that. But there are also settings where none of this really matters, yes, to be fair if that’s what you’re referring to.
> password doesn’t get sent to the server, hash does.
Kind of irrelevant what gets sent to the server and what happens on the server.
Password is already easily defeatable long before it even leaves your computer, thanks to keyloggers and/or password manager breaches and/or old-school shoulder surfing (assisted in the 2026 era by covert filming and AI).
All of which are not vulnerabilities with hardware-backed keys which you don't even need a Yubikey for these days since most modern computers and smartphones have secure enclaves and/or TPMs. (Yubikeys are still great though).
What people think of as a very strong password often has <100 "security bits" of entropy.
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
Worth noting that if you are running a remotely recent version of SSH it has everything you need built-in. No need for Fail2Ban on SSH these days if your config is fully modernised.
There is regular bot traffic attacking SSH, but it's not "saturate your downlink" level of traffic. If you're getting attacked that heavily then you're being specifically targeted.
Huh? I run ssh (OpenSSH) on port 22 with root login enabled and no fail2ban on a single-core laptop cpu from fifteen years ago. The load average is 0.00 even with the small mail and web server that runs on it
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)
Same issue with key reuse. Unless you have a policy to rotate them and verify which host they login from, someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
> someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
Even 5+ years ago, hardware-backed SSH keys were a thing (Yubikey). The only difference today is that its 100% easier to do keys properly because of the widespread commoditisation of secure enclaves in desktops and smartphones to support FIDO, its basically free of charge now because you're not obliged to buy a Yubikey.
The exploit works for a user account that has no password. This is often confused with turning off password auth, especially in containers that have only one usable account.
It's not just the parent commenter. I'm sure somebody out there is going to remove their account password instead of editing their sshd_config in an attempt to "patch" this vulnerability, actually exacerbating the problem.
Bug > are password logins still on? they are by default
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
Yeah because if attackers are already finding a way in, they can easily read my ~/.ssh/id_ed25519. A password is strictly better, although not by much because lateral movement is very easy anyways. The only exception is if the private key is actually on a HSM, which is the case for all my SSH private keys on iOS.
Your comment is self-contradictory. In one sentence you say “these days” and in another you say “never, ever.” These are contradictory. What if your computer is so old that it doesn’t have a TPM?
I was already pointing out that if you have secure enclave for your private key you should use key-based authentication. That’s what I do on iOS. Otherwise a password is better than your private key in plain text on disk. So much malware steal your private SSH keys (along with your other saved passwords and browser cookies); no malware that I know of performs social engineering and tricks you into giving up your SSH passwords.
I mean there are 65k possible ports so it's the equivalent of guessing an 16 bit password, i.e. two (extended) ascii characters. It's not very surprising that 'multiple bots' found it: a single `nmap` command will give you the open ports in a few seconds.
Never attribute to malice that which yada yada yada
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
Pretty big? For this purpose, my instinct would be to select an ASCII character that's not alphanumeric, not directional, and not a common separator, and that's like, what, a set of 11 characters?
You have considerably underestimated both how fallible humans are, and how high their tolerance is for risks which aren't immediately visible.
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
True, but also, that is exactly the pretext that would allow someone to maliciously, intentionally commit a backdoor.
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
Yes, "this looks like an underhanded c challenge" was my first thought when reading this. See one mistake by one project, inject the matching intentional mistake into another project (that also conveniently fixes some known bug), and boom - plausibly deniable backdoor.
If I had a free afternoon, I'd ask a bunch of agents to track the people responsible and their other contributions (but usually the scheme is "contribute valuable patches to build reputation, and only inject an exploit when safe to do so").
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
OpenSSH considers this to be a vulnerability in musl and not OpenSSH, but shipped a fix anyways since that helps mitigate impact... Whether you agree with that or not, it's up to you :)
I avoided using alpine-based containers ever since finding out a lot of Linux server software is actually Linux + SystemD software. The union set API of both Linux and SystemD seems to be the default "Linux API" today.
Container images rarely ship with an init service of any kind (systemd or otherwise); the expectation is that your application is the root process. This includes Alpine-based images.
That's their point. The software they want to use often assumes its being run on a platform with systemd, so running that software on Alpine which makes a point to not use systemd causes challenges for them compared to using a different distro.
Yes the other tools at the time were pretty bad, but after and around the time systemd came out, they did get much better than the sysv-init days.
Check out dinit, it's basically the same declarative philosophy as systemd, but without the bloat, and with a much simpler config API. Or s6 if you specifically care about "modular", though unless you're an embedded dev, you probably don't actually want that.
Eh, HNers are just noisier and crankier than most. The silent majority are fine with (and even like) systemd. I have no desire to go back to SysV init.
The last few weeks have made me feel very good about my policy of zero internet-accessible ssh ports, ever. On AWS, use SSM. For a VPS or home server, access everything over WireGuard.
Such solutions always felt like swapping one risk for another. With good separation (running the VPN on a separate host from the SSH login, otherwise a bug in the VPN solution still might result in a shell on the box), at best you'd need two vulnerabilities before someone gets in, but it's not fundamentally different unless you find a way simpler protocol where you can be more certain that it's secure than ssh is (for all its scrutiny and track record of no pre-auth bugs for decades)
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
One thing that makes me feel a lot more confident about exposing a WireGuard port is it’s completely invisible unless you have the host’s public key. Nothing will show up on a port scan.
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
Nice web site. However, I've got a server with 14 alpine-based docker containers in production. Among them 13 are widely distributed images (like web server, databases, etc.) and one is a custom build image. Only one of them have an SSH server publicly exposed, Forgejo, but it's its own internal SSH server which only accept public keys and Git commands. On the internal network, only 3 of respond to SSH. I haven't tweaked their default SSH config. None were vulnerable to this attack.
The bugs and commits that "fix" them reveal somewhat of a shitshow for how musl is managed. All projects have bugs, but I can't help but question why the wheel was reinvented here. Many toes have been stubbed along the way.
Did we declare "break the world Thursday" and nobody told me? I feel like it's three weeks in a row that we've had a once-in-a-decade security issue pop up.
Tangent: Pangram says the text is 60-ish% AI-generated. Not that surprising given the look of the page and the work being done with AI in the first place, but it is interesting to me because this doesn't look like your standard wall-of-text AI purple prose. So it's notable that a) you can generate convincing text in this style and b) Pangram detects it anyway.
Rust tells you off if you don’t handle Results. Sure, you can still just unwrap() it and have it panic, but at least that’s a specific decision instead of forgetting to check for NULL like in this case.
Am I the only one who is shocked that such important and security-relevant software just does unorthodox stuff like "well let's just return `*` if it fails" and other kind of "hacks" ?! The developers must know that this is very bad software engineering and will definitely cause issues at _some_ point in time, no?!
It would be nice if the passwd crypt functions didn't have timing dependence on password length. Then there'd never need to be hacky workarounds like what musl did, and various tricks to avoid exposing whether users exist on systems.
PSA for OpenWrt users: If you are getting the post quantum message on login which points to openssh.com, you are likely using dropbear nonetheless. Check in System->Software. I was confused and worried why I seemingly had the vulnerable OpenSSH installed, but its just dropbear using an OpenSSH link.
Thanks you to the team that found this. Exceptionally well explained and funny at the same time. What is most important is how people just checking stuff( remember the .xz hack that was tried?) has save the internets ass so many times. We should give good security researchers UBI for life for just doing good work.
Not the worst site, but it's clearly a publicity machine from wakefield.co, whom I haven't heard of.
Also, exploitation conditions are super narrow, albeit, any password is wild.
> except a lot of programs never check for NULL. they just compare whatever crypt() gives back with the stored field, hit the NULL, and crash
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
"unfortunately, a large portion of programs which call crypt are not
prepared for its failure and do not check that the return value is
non-null before using it. thus, always "succeeding" but giving an
unmatchable hash is reportedly a better behavior than failing on
error." [1]
Such a questionable decision! "Rather than let a bug in a caller (not handling the valid null output and thus crashing) surface, let's just silence it and pretend that the caller is correct! A potential clash in some situations (resulting in this vulnerability) is better than a crash!"
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
> did musl really feel the need to accommodate bad implementations like this
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
Am I missing something or is this written up to seem a lot worse than it actually is?
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
How many alpine installs are running sshd? Most containers using it as a base image aren't. Embedded Linux is unlikely to be running it, too. And if one IS running sshd, how often is it with unchanged defaults? I don't buy the inference here.
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
I thought Dropbear was pretty much the defacto SSH server for embedded applications, is OpenSSH+musl actually a common combination for these scenarios?
I wouldn't say the bug is only in musl. Musl is probably the "worst" offender here (at least choose something other than "*"!), but the change in sshd was also a little suspect. i.e. the "let's find a user and take their salt (and hashing algorithm) and use it to compute and compare the hash to avoid timing attacks" change.
I haven't thought it through, but I suspect in an ideal world, sshd shouldn't have accepted the user password regardless of the outcome of crypt() if the user did not exist and it only pretended to go through the motions to avoid a timing attack.
Did look into their patch yesterday(?) which seemed to mitigate this issue which they (rightfully IMHO) consider caused by musl. But it seems their fix is to do a sanity check on the supposed hashed password char length, and thus still theoretically still allowing the possibility of those "!!!INVALID_PASSWORD!!!" suggestions to fall through....
I'm genuinely curious what's the Venn diagram here because I feel a lot of embedded systems precisely have one condition but not the other, e.g. password yes but not necessarily multiple accounts, e.g root only, while using musl. Namely almost all for almost all, but if only one is not overlapping another then it's actually very few.
I thought they had a difference in meaning, i.e. * means the user has no password and ! means that password logins are disabled for this user, but the man page does not make it clear if there's a distinction.
From man 5 shadow:
> A password field which starts with an exclamation mark means that the password is locked.
> If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not be able to use a unix password to log in (but the user may log in the system by other means).
For what it's worth, default Alpine installs don't have an account with * as its password hash if you use the root account as your main one. Not sure if that's also true for ones where you set up a user, maybe those ones are vulnerable.
does this really matter? It could be used to do something devastating. The number of systems that match those requirements is only one factor. If just a handful of those are highly critical for humanity it could lead to disaster. Eg. hacks into fridges, centrifuges etc.
- blue t-shirts and green socks have been added to the default loadout for new hackers, since it's optimal according to AI and costs nothing to the attacker in terms of compute
- there are so many ley line "maps" out there that basically any server placement counts as aligned with at least two, so this one's free also
Someone owns a blue shirt/green socks clearance warehouse, this is all just and ad? I've seen weirder. I can imagine visiting SanFran in a few months, and it looks like a weird sci-fi movie scene, everyone dressed the same.
Disabling sshd password logins is security 101, and the default for some cloud provided images these days. If password logins haven't been disabled, what other doors have been left open anyway?
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
Not sure why you're getting downvoted as from security standpoint that would be the right thing to do; but actual good server providers take it up a notch and allow you to upload the public key through their admin panel during server set up and the password you get only works locally for sudo etc. There's no reason why a fresh Linux setup should ever offer a publicly available SSH server with password based auth out of the box.
Alpine Linux (which uses musl) images are popularly deployed by those who use docker, and apparently the issue is there on the default install.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
I've done it in the past with "serverless" sftp servers, which is how a lot of inter-company batch transfers happen for historical reasons, especially in the financial sector. I don't recall what images I used, but it was far easier to patch with a CI/CD pipeline. We did use keys, though.
Probably a lot of companies which have their own k8s cluster and segregate kubectl access to operations only. Developers still need a way to their app backend in some cases.
Kubernetes already comes with better solutions for that sort of thing. I could see a ssh bastion container for bridging to legacy infrastructure though. Manage its keys right and it's probably more secure than a lot of alternatives.
I have done that in the past, in order to be able to automate copying files to the container during testing or data migration. It was an ugly hack and now I need to make double sure none of those containers are still running…
Alpine in containers is usually not affected. Most images/containers haven't ssh/sshd installed at all. Remote access is usually done from the host system.
Alpine is mostly used in containers, where you don't run sshd anyway. It apparently doesn't use PAM by default though.
OpenWRT also doesn't use PAM by default apparently, and there is a risk of people using password login, which you should never do past initial setup.
But outside specific niches, musl is rare, with good reason. Performance is a joke compared to glibc, especially the allocator. The only reason to choose musl in my opinion is for memory constrained devices (OpenWRT etc) or for building static binaries where performance doesn't matter but portability does. I consider the "small container" use case questionable: performance will suffer, and you could share large base layers instead which would amortize the cost over many different images.
Yeah, OpenWRT doesn't use openssh by default - it uses dropbear. You can install openssh, if you want more features and there seem to be 2 packages - one with PAM support (openssh-server-pam) and one without it (openssh-server). PAM-less one is the one without the suffix so perhaps it's more likely you'd install that one out of the two, but only if you were trying to switch to using openssh in the first place.
> kinda like the PHP vulnerability from the other day
What? Pretty sure Cloudflare detected widespread attacks almost immediately after it was published. We are probably lucky that not much more happened because of responsible disclosure and the large media attention led to widespread admin awareness.
It’s irrelevant how widespread it is. It’s a bad bug that should be fixed. Why do people always say stuff like this? If you’re running a system that is exploitable then you deserve to know about it and to be provided a fix. It doesn’t matter all that much how widespread it is. The CVE system doesn’t exist to cater only to highly exposed systems.
It matters how widespread it is because humans have limited time and resources to solve problems. If these conditions only exist in 0.001% of installs, is it worth the man hours to fix it? Alpine may think not, because like others have mentioned, there are likely other higher probability vulnerabilities being exposed in order to meet the conditions for this to even matter. A judgement call can (and clearly has?) be made to prioritize other work.
The good news is that considering Alpine is open-source, you can be the one to make the judgement call that it is worth fixing and go out there and fix it ;)
A trivial-to-exploit auth bypass is always worth fixing, no matter how few systems are affected right now in practice. My question is whether it's worth giving the vulnerability a name and making a webpage for it.
0.001% of all Linux installs (including containers, CI workers, embedded, etc) is still a huge huge number. It should be taken seriously until proven otherwise
What this tells me is that musl+openssh-server is an incredibly uncommon setup, as "at least 257 character string" is literally the third thing I try after "admin" and "12345" when greeted with a password input.
This is a super duper annoying site for people who just want to know if their systems are vulnerable, after patching like a dozen critical vulnerabilities in other software over the last week and are almost at the breaking point of resigning.
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Thanks for mentioning that before I tried to scroll down. Forewarned, I was able to click on the "2026" link once the left-handed "scrollbar" (not the right word but I can't figure out what the right word should be) appeared, which let me jump past most of the verbiage and get to the "are you affected?" section just a couple pages later.
I'm starting to realize that trusting some random people who wrote the gruddiest, boringest, most torturous low-level stuff might have been a bad plan.
At this point I feel embarrassed in front of the AIs.
Can we just have a site with the text and images without the awkward scrolling crap? Hijacking basic features is an instant click away from sites for me
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
This AI slop write-up is so fucking annoying to read. I don't even mind the AI generated website. Why can't you even bother to spend like an hour to properly sum up the exploit yourself?
It's shocking that it took so long with the amount of investment and hype you'd expect this kind of bugs to be already fixed, but it still need a human to tell the agents what to do, which could also work with any fuzzing framework
something interesting to note here is that while this isn't a memory safety vulnerability, a modern systems programming language with better error handling could have easily avoided this whole situation. error handling in C is a PITA, and we're paying the cost of decades of pretending that it's not a problem
It’s dumb and negligent to be missing “PasswordAuthentication no” in 2026. It’s been so for at least ten years. OpenSSH should have turned off password auth in the defaults years ago.
All that site is lacking is marquee and furry creatures. AI slop sites are so horrible with their blinking animations, l33t dark background and chaotic text.
what was done to report & patch the vuln? Why publish it in this format? It’s presented like a whistleblower who’s exhausted every approach, but the effort to do the right thing is not clear.
Yep, scrolling through my phone and reading slowly to understand. I'm seeing distracting animations below what I'm reading. And when I go to them, there is no way to restart them. So it completely misses the point.
I think it varies from person to person. Since I'm kind of from the TikTok generation, I don't have much trouble reading this kind of animation, but some people prefer static sites. That's just how it is.
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
There is a myth that TikTok generation has ADHD and can't concentrate, if it were true I'd expect static sites would be easier for you, like to me. I mean, I struggle sometimes to keep attention and not to wander off mentally. With static text it is easier to fix, you have just to jump back a bit, when you notice that you've forgot to read while you were reading. Or when you are lost at too many trees you can kind of look at the forest, at the text as a whole and find your place there. But with all this scrolling it is hard to jump backward/forward at will. It feels tiring and annoying.
> how did I ran into it? [...] i had agent swarms. a lot of them. let's not get into it
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
Not sure why but a lot of people seem to be missing that you don't have to look at AI as a binary "you're either with us or you're with the terrorists."
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
> Can we please now normalize AI usage in development?
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
I feel like there is a lot of pushback from delusional engineers, that won't accept their coding skills have been rendered useless - so they come up with tons of reasons to avoid AI, because it just has to be bad. Anything to hold up the ilusion you are an well-paid white-collar worker that can pick employers anytime. And I say this myself coding since the late 90s and making a living off it - it is a painful pill to swallow, but the genie does not go back in the box so just accept it and move on.
Can we hear something from those who say that using port-knocking is "security through obscurity"?
Really: from the XZ backdoor in SSH (which went live on some distros) to this...
What's you rationale for calling port-knocking "security through obscurity" and not "defense in depth"?
Put it this way: if I advertize that my server is using port-knocking, that the sequence changes every 30 seconds (like TOTP) and that as soon one sequence is used, it's unusable (so you've only got one attempt to SSH in every 30 seconds, even if you know the sequence)...
Where's the security through obscurity?
But really: even if had musl and PasswordAuthentication allowed (I don't have neither), how would an attacker gets through the port knocking?
More fundamentally the question is: in which way is your system that does not use port knocking more secure than the one who do?
(remember: we're talking about remote exploits by simply having exposed TCP ports open or SSH listening)
Wanna have a talk too about the "solutions" proposed by "security experts" here "because SSH is too old" and that happen to be crufts on top of SSH that had remote root exploitable backdoors? (sorry, holes)
> on alpine's official aws and azure images that's the default alpine user, and it has passwordless doas. so it's root
good fucking god. if I get this correctly... you can insta-own any aws/azure instance that was set up with alpine as long as it has ssh enabled, which it probably has.
Disclosure: I work at Detectify. A few practical notes for anyone checking their own machines:
- It affects Alpine's openssh-server (built without PAM), or openssh-server-pam with UsePAM no, when PasswordAuthentication is enabled. Alpine says every release they checked is affected, 3.16 through 3.24 and edge.
- Fixed openssh packages are out for all of those, including the older 3.16-3.20 releases. Upgrade and restart sshd (rc-service sshd restart). If you can't upgrade yet, set PasswordAuthentication no.
- To check a box: apk info -v openssh-server for the version, and sshd -T | grep -iE 'usepam|passwordauthentication' for the effective config.
- Containers built FROM alpine that run sshd are easy to miss in an inventory. So are the official tiny-cloud images (3.18.11 through 3.24.3), which ship an alpine user with no password and passwordless doas, so a bypass there means root.
- After patching, look in /var/log/messages for sshd lines saying "Accepted password for" an account that has no password.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
It is the opposite that is unsafe, that is, when the returned value can't be NULL, but it stills returns a pointer. C++ has references in an attempt to fix this problem.
Because C tends to prioritize performance over safety, a NULL pointer dereference is considered undefined behavior and the compiler can invoke nasal demons if it makes the code go faster, but with the right options, you can have it checked and invoke what in other languages would call a panic handler, but on UNIX is the SIGSEGV signal. You can then attach a debugger or read the core file and see what went wrong.
Note: I am not talking about "char *" for strings, that one is unsafe, but NULL is not. It is just a special case of an option type, or a result type, or an error union. Modern languages just extended this idea to objects other than pointers, and with more than one type of error.
"Make invalid states unrepresentable" is such a useful adage when designing even basic systems, much less the gates to control your server.
Maybe I'm wrong, but it looks to me like the exact same incentives and means to screw things up exist in C and in Rust for this particular bug.
For example, in a Rust web app, you can choose to use String for encoded URLs, unencoded URL fragments, escaped html, unescaped HTML, usernames, etc. Or you can choose to use newtypes and carefully design all APIs to only take and return the correct newtypes.
The problem is not that you can write bad stuff in rust, the problem is that this particular usage pattern can't be sanely coded in C. This is a public API that should inform users and enforce certain properties. A Results type does exactly that, you would have to go way out of your way to force invalid stuff through that type alone.
And that just ends up indicting the C language itself. Can't send errors down a separate channel; folks might not inspect errno. Can't return NULL on error, folks might compare it blindly to another result. No tagged unions, so can't return one. Can't crash on error because there's no scoped error handling. Can't return a struct for checking .error/.result separately, because alignment and equality (and tearing and accidental clobbering and more) make that brittle as hell for widely-used APIs.
C is not a good language for library APIs. Like, really not. It has its uses/strengths, sure, but without even getting into the typical gripes people have about C, libraries--one of the things that it is used for the most--are one of the things for which it is the least well-suited.
"But it has a stable ABI!" So does a land mine.
The surface-level hints are gone, but the attempt to make every sentence pithy, the decisions on what information to present and what to assume the audience knows, and the over-emotionalization are all still there.
In the past I could get an explanation, where I was able to track the line of thought. Now when I read through the explanation it is like someone has taken two or three explanations and merged them into each other, like there is a combining of iterations without clarity.
example:
The tree is blue.
The tall tree has the color of blue.
the tree, while tall, is blue.
That finally gets squished into: the blue tall tree has height and is evident of blue.
Is this double speak or injected double effort of words?
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
Except it turns out that at least one of those things is because I didn't enable JS, and the static text is wrong!
(Turning on JS did not actually help, because then the page is filled with horrible, cheesy animations and I can't actually read any of it.)
wtaf ?
musl switched to returning `*` so folks stopped accidentally using NULL results, which was causing crashes and risked false-positives comparing NULL to itself.
That was a misfeature and should never have been implemented. But I bet it was an easy band-aid change to make!
Of all the things to mitigate...
1) crashing should be an option - it noisily indicates "there's a problem" and if it faults and dumps core, you've got a guided tour of your problem.
2) Is testing NULL as a passable option likely? I'd guess a more likely idiom is:
> The behavior of crypt on errors isn't well standardized. Some implementations simply can't fail (except by crashing the program), others return a null pointer or a fixed string. Most implementations don't set errno, but some do. POSIX specifies returning a null pointer and setting errno, but it defines only one possible error, ENOSYS, in the case where crypt is not supported at all.
In my opinion the problem is that the behavior of crypt is underspecified - it isn't even specified how it signals failure, which is shocking.
Another disturbing decision that crypt's signature takes in the hash its output is supposed to be compared against - if it weren't for that, this half-baked "on error the output won't compare equal to the input" measure that crypt and crypt_* use wouldn't have been possible.
You can't check errno to see if there was a failure because, as the man page says, the value of errno is unspecified when crypt succeeds.
Why * specifically? Why not some random guid generated before committing, if it's so important to not let buggy software crash?
But buggy software might not crash; it might compare NULL == NULL and misbehave, instead.
So there's really no good option here, which is why C is a rancid choice of language for publicly-usable APIs. But we're stuck with it. So it goes.
But issues caused by folks forgetting to check errno, miscomparing NULL to NULL when calling multiple APIs, dereferencing external-API-returned NULL due to missed error checking, mishandling inherited errno after successful action, or--the most common--mishandling negative integer return sentinels meant to indicate errors? Those are incredibly common.
Standard C api calls return an integer, which is 0 on success or nonzero on error. Ignoring it is an intentional choice. `errno` is the hack that shouldn't really exist because it's trivially easy to forget to check that and also has issues with threading (patched by making it a thread local) and more recently, coroutines (which admit no general purpose patch because they're a userspace construct).
Another reason why this is terrible is that it's unclear what the returned pointer's lifetime is. Is it until the next call to crypt? If so, is it thread-local like errno? If it's passed on to the user, is it safe to call free() on it, or is there a crypt_free function that should be called instead?
But all of this, and I mean all of it pales in comparison to returning '*' on error.
100% agree. This is an egregious failure. But C's error-handling story would be bad even without this, and the badness of C's error handling is what motivated the (bad) decision to return *.
It'd be great if we wrote systems software in a language that supported advanced features like...
checks notes
...returning multiple values.
Hell, even in VBA you had to at least spend the keystrokes to type "on error resume next".
So they made the (very bad, but motivated by C's broken design/conventions) decision to return a magic string instead.
So the knock-on effect is that libraries that do have errors other than ENOSYS to accommodate, have to return magic 'impossible' values instead of setting errno and returning NULL, because of the weight of existing applications that expect nothing but the standard, traditional AT&T Unix (and indeed old BSD), behaviour.
Check the GNU libc crypt() doco. It defines a whole family of 'impossible' values that signal errors, by dint of listing a load of magic characters that should never be in the encrypted string. Including asterisk.
Also check FreeBSD's OpenPAM for a case in point. Even today, it does not check for a NULL return from crypt() and leaps straight to strcmp().
* https://cgit.freebsd.org/src/tree/lib/libpam/modules/pam_uni...
If you want to see real history here, check out the 1992 comment by Eric Andrew Young, that is still in the source for OpenBSD today (in des/des_fcrypt.c), on how SSLeay's crypt() had to be adjusted so that any single character password field, including "*", did not end up successfully matching itself.
* https://cvsweb.openbsd.org/annotate/src/lib/libcrypto/des/At...
It's worth pointing out that musl tried to do the same as SSLeay, and deliberately returns something different to the original salt so that they will not compare equal. It does not return "*" if the input is actually "*" in the first place, contrary to the headlined cartoon explanation. (The locus of the problem is the caller calling it with a salt that is not "*" and then comparing it with "*".)
There is simply no universe in which a security-conscientious reviewer looks at the code, spots * being a valid return value, remembers that * is the default value in /etc/shadow for "no password login please" and then is all like:
This is fine, you check for * and return + instead.
There are more 32-long character sequences than atoms in the universe. Pick one, any of them instead.
Of course given the chance that one might do a direct string comparison between crypt's return and the passwd/shadow file's hash field it was still a not well thought out choice.
OpenSSL??
[1] https://en.wikipedia.org/wiki/In-band_signaling#Computer_dat...
So, any field de-reference in Java can produce a null pointer exception. Try testing all paths in something like:
try { foo.sideEffect1().sideEffect2()...sideEffectN(); } catch { bar.sideEffect1().sideEffect2()...sideEffectM(); } finally { quuz.sideEffect1().sideEffect2()...sideEffectP(); }
There are M * N * P paths. The only way to write testable code with exceptions is to panic on uncaught exception, and to never catch unexpected exceptions. Even then, you're relying on nothing unexpectedly throwing the exception type you caught.
I'd argue Result<T> >> NULL >> sentries >> exceptions for this reason. Sentries are terrible, but at least they're not computationally hard to use correctly.
And I'm not a fan of breaking control flow, but I still find it preferable to nulls. A null may travel far from the place where it was generated before blowing up, making troubleshooting harder, or it might come from a function you called for its side effect but forgot to check the return value. I gotta say I'm also a careful enthusiast of checked exceptions, it's great knowing which ways a function can fail.
But I think the consensus is growing that Result<T> (plus warnings for unused return values) is simply the superior choice.
Which is why the pattern of double-checked locking doesn't work.
https://en.wikipedia.org/wiki/Double-checked_locking
That can only happen if you leak the reference (given we are talking about java language level semantics and not byte code), which I give you is unfortunately not that hard to do.
As for exceptions, I think they would pair beautifully with software transactional memory, which would work well with effects. In that case (and no side effects) it would not matter what code path we took, we could just go back to a known safe point.
Because ergonomy-wise I still think exceptions win (even though I do like Maybe types, and Rust's ? macro is plenty good): - they let one focus on the actual happy path code - no error is ever swallowed, unless explicitly written so (looking at you go with useless if errs that fail to properly handle the error), they are always handled somewhere - they automatically "unwrap" the Ok result, and automatically bubble up the Err side. - bubbling up is especially useful because most errors can't be handled at their source (e.g. IO error is meaningless, how you handle it depends on what you want to do in the first place, which is done several layers higher) - they automatically attach a stackframe/source, while with result types you may get a list of Ok Ok Ok Err and have no idea how that Err got there.
Yes, broad exception catch clauses are simply incorrect code.
Panic is maybe an overreaction, "scratch everything and orderly abort" is sufficient.
So yeah, I guess it either needs to happen at runtime, or we just do signaling and user data on separate file descriptors.
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
https://github.com/benbernard/RecordStream
I have a bunch of management scripts, etc, that need to run on everything from a potato to a T-1000, and that's the only language that's always available. (OpenBSD sh, FreeBSD sh, busybox, bash, dash, zsh all can cope with it).
I'd love something that's at least JSON-typed (or equivalent), but that can be bootstrapped up on least-common-denominator posix.
Has anyone built something similar? (Yes, I'm being picky -- bun / ts is already a pretty low bar for cross platform support, but these scripts do stuff like install node, so I'm not totally crazy for asking.)
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
https://pubs.opengroup.org/onlinepubs/009604599/functions/cr...
>Upon successful completion, crypt() shall return a pointer to the encoded string. The first two characters of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
OP's sshhole link specifically notes that it _should_ return NULL, but some MUSL devs decided they wanted it to return trash instead, and the trash they returned instead of NULL caused this.
All because some devs were too lazy to check the result and another dev decided to be clever and "fix" NULL errors from crypt by sending back 'unmatchable' trash banking on devs not checking NULL.
edit: apparently crypt implementations have been trash ignoring the NULL return for a long time. this isn't only on MUSL, but it's still because people ignored and tried to work around the existing out-of-band error mechanism.
But all those sentinels were added for the reason that, when your code is a public API intended to be widely used by who-knows-who, it needs to be really difficult to misuse it. NULL-plus-errno is easy to misuse. Yes, it's a dangerous mistake and your own fault if you do it. But people have been making that mistake for years and don't show any signs of stopping.
The root cause here is that C is fundamentally a rotten choice for public APIs of any kind. I griped more here: https://news.ycombinator.com/item?id=50381800
> OP's sshhole link specifically notes that it _should_ return NULL
I meant there's no mechanism besides returning NULL that would make it hard to ignore the issue (exceptions need to be caught, results need to be unwrapped and their failure handled, an integer return value makes it clearer that there's something there that you need to check). I also missed errno codes when glancing at the manpage earlier which should illustrate the problem.
And yes, whoever first had the brilliant idea to start returning garbage never should've done that, I'm not disputing that.
No error handling scheme can fix that problem. For instance, if a Rust API says I should return a Result<String, ENOSYS>, I can always write code like `if(ENOSYS) { return Ok("*") } instead of returning Err(ENOSYS).
Then my code would be compatible with MUSL.
The only reason this library did the insane thing of returning "*" is because it was extremely easy to cause crashes or false-positives (NULL == NULL) by returning NULL+errno. The reason that was easy is because C lacks a means of making error handling mandatory (exceptions, tuple returns, tagged unions, whatever).
If you return a tagged union, users might mis-handle it, it's true (https://blog.cloudflare.com/18-november-2025-outage/). But they have to handle it, which is a requirement that C APIs are incapable of providing.
Also, in things like rust you can just do result.unwrap() if you want. In general, language constructs don't really save you from bad programmers.
In practice, that's less of a risk than:
Or code for which pointer identity is relevant (e.g. accessing slices/subparts of a data structure): And for sure there are much better ways to write both kinds of code! But people routinely do not do that.You can do that but it's going to feel bad to write that and it's going to stand out like a sore thumb to anyone who reads it. But "errno" is just ambient state that's completely invisible unless you happen to have the manpage fresh in your memory.
Here's how to explain it in order to blame openssh:
openssh processes passwords and doesn't want to reveal through timing whether a user has a valid password (or exists at all). So far so good.
What they do if they hit a user with no valid password (i.e. "*") is that they look for another user with a valid password, to use the seed from. A bit weird already, but OK from a perspective of avoiding implementation specifics on seeds and hash algorithms. They know at this point that the result must be to deny access even if the hashed value matches, because that hash is for another user.
So they put the password received over ssh into crypt() with this other user's hashed value to get a seed. And then accept it if it matches, apparently?
Homework: what happens if the password actually matches another user? Can I log in as root if openssh randomly picks my user for the fake salt, and I type my own password correctly?
Also I think it could have returned ":". Given the file format of /etc/shadow and /etc/passwd I don't think it's possible to have a : as a value for anything.
I wonder if somebody else would have thought the exact same thing in a different (non-/etc/*) context, and then you get the same effect in a more obscure manner.
Some badly designed password hashing algorithms (cough bcrypt cough) have maximum password lengths, ignoring everything after X characters. So it could happen that I set my password to a 100-character string, it gets saved as "==DANGEROUS_CRYPT_ERROR_length==", and then anybody can log in to my account by entering an unrelated long string that returns the same "error".
If a function promises a string result, it should only provide a string if the function succeeded. The null was much better than magic strings.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
For instance, what if your error handling code throws?
One spectacular example of panic saving the day was the Apollo 11 1202 error. It was basically a watchdog timer that fired if the landing computer was starved of CPU time. A misconfiguration caused the first moon landing to hit CPU overload. When that happened, the computer displayed 1202 and rebooted, then ran things in priority order. Apollo 11 landed in the middle of a tight 'panic, reboot, adjust thrusters' loop. Thanks to core memory, most of the heap survived the reboot, so the reboot step was fast enough for it to mostly land without human interaction. The pilot had to make ~1 adjustment to get it to actually hit the landing site, but the panic-ridden computer handled the rest of the descent.
https://dpma.de/english/our_office/publications/ingeniouswom...
Panic is fine, but fundamentally a whole-program crash is undesirable in a really wide range of use-cases. I don't want other SSH connections to crash (or the server to stop accepting valid logins until someone restarts sshd) if one person tries to log in with a credential that triggers a panicking bug in crypt(). Same for crashes in request handlers and so on.
The language/runtime itself doesn't (and probably cannot, at least if it's a general purpose language) distinguish between local and global error conditions, so aborts end up being either unnecessarily damaging, or languages like Go end up normalizing panic "localization"/handling in dangerous ways. Isolation using threads isn't robust enough. Isolation using processes is far too fiddly, slow, and opaque.
The better solution is required error handling. If user code must engage with the possibility of an error state in order to access a result, that's a huge step forward. They can do the error handling wrong, but if you make most people have to do it at all most of the problems will go away.
There are other good solutions too, like Erlang's process boundaries, but those tend to have drawbacks that make them inapplicable to a lot of programming areas. Tagged unions, on the other hand, are only inapplicable in C library code because of language defects and terrible community conventions around ABIs.
"I forgot to check for an error" should not be implicated in bugs in 2026; we have had a large menu of possible alternatives (checked exceptions, unchecked exceptions, tagged unions, etc.) for decades. But we're still over here writing C (and Go), languages pathologically incapable of providing this ability.
This is also why you can upgrade SSH over SSH without having the connection drop mid-stream.
Again: You cannot avoid "I forgot to check for an error". What happens if you get a sigpipe on stdout when printing the "I crashed" line? If Linux overcommits memory? If you turn off overcommit, port all software to check for + cleanly handle NULL from malloc (a substantial effort!), but then get a machine check exception from a double bit ECC error during an infallible read?
It has to: if it did 100% of the post-accept(2) work in a subprocess it would be pretty easy to remotely forkbomb/PID-exhaust a system by spamming invalid logins.
> You cannot avoid "I forgot to check for an error"
Yes you can. Checked exceptions and tagged unions both force calling code to be written with error-awareness.
In SSH's case, "I forgot to check" might lead to crashes (which might knock over sshd so new connections are impossible), or it might lead to false-positives via comparing NULL == NULL.
I'd much rather that boneheaded API users have to write code like
At least that way some keystrokes are required to do the stupid thing. People still will (https://www.schneier.com/blog/archives/2008/05/random_number...), but the harder we can make it, the better.We have seriously overused strings and we're now paying the price.
I can't even get an obvious UX improvement through PAM because they like it broken.
https://github.com/linux-pam/linux-pam/issues/778
Honestly I've been shaking my fist at these since 0 becomes True/Success in Bash and the opposite everywhere else
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
In a sense, they should've changed the parsing logic to return a struct with a field 'password_hash: Option<String>', and defined the grammar of the field to be: "a hash string, or '*' for no login". The bug is, in part, caused by the haphazard way they don't change any formal grammar or contract and just relied on the fact that '' happens to* never be a valid hash.
It's the difference between "happens to work" and "defined to work by construction". The former approach breaks once a hash function is invented where '*' is a legal hash (such as this new musl crypt() hash function where all strings longer than 256 bytes hash to '*'). The latter keeps working regardless because the asterisk is treated as "no password" already at the parser stage.
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_des...
(hey, this workaround is wrong if the salt is "x"!)
but they forgot to work around it here:
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_sha...
The vulnerable path in musl is only a little bit vulnerable, because "*" is not actually a sha256 encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
https://github.com/openssh/openssh-portable/blob/master/auth...
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
When the salt is "x", it'll return "*" and still not match, won't it? The code is correct as long as the second argument is the hash that's being checked. And like you say, the sha512 path isn't relevant since you only get there if the salt started with "$".
> Turns out that sshd detects that the password hash is too short and does not forward the salt!
I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
[1] https://lists.mindrot.org/pipermail/openssh-commits/2016-Jul...
I think you're right. That's what I get for doing a very very quick read.
> I think you're misunderstanding the intent of that check. It was added by the 2016 change linked in the writeup[1] and causes OpenSSH's xcrypt() to select and use a salt from a different line of /etc/shadow to prevent timing attacks. It is correct for a crypt() implementation that returns NULL on error.
I don't care what the intent of the check is. If you're doing a fake password hash to obscure timing, remember that it's fake and don't let the user log in! Don't deliberately pass nonsense into the live check and rely on the nonsense check failing to prevent the fake check from accidentally working. And don't have a code path that thinks it's checking the real password but passes in the wrong second parameter to crypt().
Or, better yet, have an entirely separate function fake_check_password() that does the fake check, returns void, and cannot possibly be confused with the actual password check code.
That would be a much better approach. I see your point now.
Don't return empty string/asterisk/anything else. Instead, return a failure value.
When absolutely needed, choose the magic string with more caution:
One string showing the SHA-256 of a non-existing hash? It's a good idea for it to be a valid hash. Now we have a new risk: if it's broken once, it's broken for all. So it's a good idea for it to be a random one for each machine and/or account. Even this might have problems I can't think of, but it seems to be as secure as an account with a real password.
If this is not a good decision for some reason, choose something unmistakable: "--ACCOUNT-HAS-NO-PASSWORD--". This is much less likely to be mistakenly used in a different context, is much easier to look up in a code base than "*", and is self-documenting.
My AI usage involves telling it to do things and setting up lint rules to catch all the bad programming it’s learnt.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
Granted I mostly work with high level programming languages so I may be missing something.
But that's not a perfectly valid argument, and it never was.
The whole thing exists because APIs that return a NULL value that "nobody checks", plus a string that you shouldn't use if you actually did check the null value.
In languages with sum types, this cannot happen.
Either you get your hash, or no result at all. You couldn't get eg NULL and *.
But there has to be something in the payload field when it is not applicable. A null is perfectly fine.
The argument for changing that to "*", if I understand the above linked discussion correctly, was that some users of this api threw away errors and a dummy value would be harder to ignore. Maybe one could argue that these behaviours are less common in Rust or Go, but not because of a technical difference.
The world is not perfect and sometimes you really need to work around bad users. Even in Rust or Go. I am not convinced this was such a case. If you are convinced it was it would still have been better to select a less trivial dummy value (such as "*:ERROR:*").
Not in languages with ergonomic sum types. A `Result` or `Either` will have no payload field on the error type. The compiler won't let you dereference it.
I can think of two common patterns that I've seen frequently:
* Return NULL for errors (sometimes with a global errorno() function).
* Return an integer error with any return values as out parameters.
So if it is possible to improve, why doesn't anyone do it?
Because in fact all the changes were about changing the function signatures. The function signature is enforced by the linker. The semantic of strings that go through isn't.
Returning 2 values is possible in C (via pointers input parameters). The main issue here is that libc is a strong API contract based on C function signatures, that nobody can change, only augment with new symbols. Such strong API compatibility constraints happen in any programming language, and workaround to fit new behaviours in old APIs happens everywhere.
But also, its not like there were many alternative languages when the POSIX standards were designed so it's fair that it works like this.
- sum/union types
- multiple return values/tuple types
- exceptions
means that most ways of returning an error from a function are not available in C. Because of this, many C APIs, including `crypt` in this case, resort to in-band error signalling by using a sentinel value (like NULL, a negative integer, "*" etc.).
Most other languages lack one or two of the above, but very few lack all three, and so most have developed a different culture around error reporting (Result in Rust, multiple return values in Go, exceptions in Java, Python etc.).
And really often when your "in-band error signalling" in Rust is more expensive you realise (perhaps to your horror) that your C wouldn't be correct, so you just dodged a bullet. For example Option<OwnedFd> is, like OwnedFd, a 32-bit value, Unix has always reserved -1 to mean invalid, so Rust can treat that as a niche. But the Windows equivalent Option<OwnedHandle> can't do this because, well, cue Raymond Chen: https://devblogs.microsoft.com/oldnewthing/20040302-00/?p=40...
But aside from NonZero- and NonNull-based layout optimizations, I don't think Rust is doing anything special to obtain that machine code - it's just relying on LLVM to optimize things.
However, I also think "weakly typed" is not a well-defined concept, everyone seems to apply it differently. In particular, it seems to mostly mean "playing it loose with types in a way that I don't like".
Implicit conversion could be seen as weak typing, but it is mostly limited to specific arithmetic operations in C. I would see implicit conversions as problematic if they allowed operations that make no sense. Decay of arrays and function to pointer conversion also do not seem problematic to me for this reason.
Void pointers are also weakly typed, but they are specifically designed for this. So if you do not want this behavior, one would simply not use them. It is a bit like using "unsafe" in Rust.
There are certainly stronger typed languages, but on the full range C is more on the strongly than weakly typed side in my opinion. A language that allows arbitrary implicit conversions between pointers, would be weakly typed, or which allows calling functions with wrongly typed arguments (which was true in the past for C a long time ago)
And it's very evident why the difference matters. People gravitate to what's readily available and ergonomic. That's why every C API under the sun (most definitely including libc itself) does janky stuff like returning -1 or "*" on error instead of building their own bespoke Result-style abstraction. A programming language is not just the things that are in some way possible within it, but more importantly the things that are easy in it.
A language should encourage writing robust/secure code, not just make it possible.
In any case, I do not think the issue here is caused by not having a pre-defined result type. Checking for null would have been fine. The issue was that to accommodate old and broken code a compromise was made. But as long such compromises are done even where fixing such old code would be quite easy, it is clear that changing to another language is completely unrealistic. So what are we even talking about? The real issue is that nobody wants to spend money or effort maintaining the old code. No language can fix this.
In any case, you can not change the API of an existing function. But returning NULL for error has required by POSIX would have been just fine.
While not mandatory, a person would probably use it instead of the * hack.
Simply because the calling side wouldn't have an option to forget the check like with the NULL case
- result.unwrap() will panic if it failed
- result.ok() throws away the error but gives you a None value instead (still can’t use it as if it’s a success)
- matching on the Ok variant and doing nothing on the Err variant, you only get to use the success value inside the Ok arm
Etc etc.
The problem is when the compiler says “NULL is a valid instance of T”, which go lets you do, and rust doesn’t (without using the unsafe keyword and intentionally casting a null pointer, which you don’t do by accident.) Go lets you write code like `var hash string; hash, _ = hashfn()` and it compiles and now you have null instead of a valid hash.
This by the way is Tony Hoare's famous "Billion Dollar Mistake"
And why because some dude was complaining sshd or whatever was crashing because it didn't handle nulls? Just handle the null my guy you already know why your bullshit is crashing.
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
And now we have this.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
Story time?
100%
Go ahead and program a flight controller this way, I'm sure nothing bad will happen.
Hey, but at least it can’t touch the System32 folder.
Great job guys.
Objective-C ahem nil messaging...
Sadly, we will never learn this because failing fast runs counter to intuition, and a class of person will never override it.
Am I doing anything wrong here?
OpenSSH_10.3p1, OpenSSL 3.5.9 29 Sep 2026
PRETTY_NAME="Alpine Linux v3.24"
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
https://pkgs.alpinelinux.org/package/v3.16/main/x86_64/opens... Build time 2026-11-05 06:53:46
on 3.21 with the previous version ...
Didn't notice that the openssh-server-10.3_p1-r2 was released today.
I only ever see Alpine Linux used for containers which don't need accounts with interactive user logins, which is not to say it doesn't exist, just that it doesn't seem like Alpine comes with all preconditions met out of the box
> accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
But yeah, Alpine for persistant installs isn't unheard of - just not as popular as the container use case. They offer official VM, Xen, Pi, and bare metal images users deploy as well. I actually had a fleet of VMs to update!
Edit: The Alpine notice made it clearer to me that at least one other account has to have a password, it's just the target account that doesn't. So if you truly have no users with passwords in your container deployment then those passwordless users aren't vulnerable to this bug, but any of the image types which have a user will likely have root exposed.
> The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash.
https://alpinelinux.org/posts/2026-11-05-openssh-password-au...
Also didn't have much luck on Alpine 3.20.10 with OpenSSH_9.7p1 which I've freshly installed from an official ISO.
Last two entries in /etc/shadow:
SSHing into test@ with one 'wrong' password then one 'long' password doesn't work. I have UsePAM no and PasswordAuthentication yes in my sshd_config.I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
0: https://man.openbsd.org/sshd_config#PermitRootLogin
1: https://cvsweb.openbsd.org/diff/src/usr.bin/ssh/sshd_config?...
- an account with /etc/shadow hash set to '*'
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
Do you have an example? Looking at Alpine's advisory, I can only see that their cloud images (i.e. what you use to run a VM, not a container) have a default alpine user, no container images. And even with those, cloud images with cloud-init are unaffected with default config due to `PasswordAuthentication no` being inserted by cloud-init into sshd_config. So that leaves... Cloud images that use Tiny Cloud for provisioning. That's an instance bootstrapper authored by Alpine from the looks of it so I'm not sure how many people would prefer that over cloud-init (which a lot of VM/bare metal management software may already make easy to use) or just a non-cloud image.
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
This was not security testing as such - mostly latency, performance and hardcore network "can we break it" and "does it comply" stuff, but also a lot of cable and module / line card pulls, pluggable and fiber breaks, random reboots and pre-empting a lot of possible operator inventiveness. At one point we considered SEU bit flip testing on rented cyclotron time.
And always we battled against the top brass urging us to speed up the process and don't bother with "really basic" testing, because we observed software quality going down over the years as the push for release and time to market stared to dominate.
So we always started with sanity testing - and that included oversized password and CLI input strings. Always. And we caught things more than once, but never like this.
We were right after all.
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
Are these people mad? Why on Earth would anybody indicate error like this?
> mostly the boring parts where it talks to libc, like the password code
Is clearly not human.
This might be the beginning of the reason we shouldn't be reading the code. It's like in chess engines where the players complain about how weird, alien and unintuitive the moves are. But 100 steps down the line the engine finds some edge case exploit that gives it enough advantage to win.
So we may be crossing a line, right now, where the non-intuitive alien code being generated by agents is seeing edge cases down the line that might allow exploits. And if we "correct" the agent, we may be leaving ourselves open to exploits we don't see and have difficulty understanding.
This thread highlights why I often say that at some point in the future, the obsession with whether something was written by AI or not will be classified as a mental disorder.
It is tiring to virtually every day see this discussion play out in some HN thread or other. It's also just noise. I don't think your comment, nor the thread it inspired, added anything of value to the discussion.
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
[1] https://web.archive.org/web/20101227181948/http://aluigi.alt... [2] https://web.archive.org/web/20060522175643/http://aluigi.alt...
gregor@krekrm3 ~ % ssh root@192.168.1.1 root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Permission denied, please try again. root@192.168.1.1's password: Connection closed by 192.168.1.1 port 22
As in: "Actually out there"
The hypothetical Alpine exists, yes, but what runs that like that?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
There's no firewall and no password. Hack away HN. cd /pubget so*
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
It would be super helpful if you would answer a "what, specifically" question with a "this, specifically" itemized list or not answer at all.
This is not helpful.
Dropbear is unaffected.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
I feel the opposite, note how most all AI vulns discovered use source code level analysis, empirically keeping source code hidden is more secure.
kerchoff’s principle is misconstrued by a voyeuristic internet that wants to see source code
(almost) No company actually has secret source code, only source code not publicly viewable. And the LLMs crawling those don't belong to people who make public disclosures.
> You can't be serious with your comment. \n\n No its not.
If you differ in opinion, maybe provide evidence? Especially for a claim like long randomly generated value 1 being absurdly ("can't be serious"-level) weak as compared to long randomly generated value 2
If you mean it's log(62^20,2)≈119 bits of entropy instead of 128+ and so it's theoretically weaker when we build a Dyson sphere around the sun to brute force that key (after somehow obtaining the shadow file, since an online attack would be way too slow), then sure
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
SSH certificates are cool too.
but I guess that the hash is always the same unlike with pub priv key mechanism.
Kind of irrelevant what gets sent to the server and what happens on the server.
Password is already easily defeatable long before it even leaves your computer, thanks to keyloggers and/or password manager breaches and/or old-school shoulder surfing (assisted in the 2026 era by covert filming and AI).
All of which are not vulnerabilities with hardware-backed keys which you don't even need a Yubikey for these days since most modern computers and smartphones have secure enclaves and/or TPMs. (Yubikeys are still great though).
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
The best password in the world will still be defeated by a simple keylogger or breach of your password manager.
Worth noting that if you are running a remotely recent version of SSH it has everything you need built-in. No need for Fail2Ban on SSH these days if your config is fully modernised.
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)
Even 5+ years ago, hardware-backed SSH keys were a thing (Yubikey). The only difference today is that its 100% easier to do keys properly because of the widespread commoditisation of secure enclaves in desktops and smartphones to support FIDO, its basically free of charge now because you're not obliged to buy a Yubikey.
It's not just the parent commenter. I'm sure somebody out there is going to remove their account password instead of editing their sshd_config in an attempt to "patch" this vulnerability, actually exacerbating the problem.
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
Does it ?
Somebody here has already said "You can set PasswordAuthentication=no or update to remediate"
We don't use musl at $work anyway, so I've no means to test.
Why not ?
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
No excuse for doing that these days.
You can use a Yubikey or you can use your computer's secure enclave or TPM with FIDO at no extra cost.
A password is never, ever "strictly better". Seriously, you're just making shit up now.
I was already pointing out that if you have secure enclave for your private key you should use key-based authentication. That’s what I do on iOS. Otherwise a password is better than your private key in plain text on disk. So much malware steal your private SSH keys (along with your other saved passwords and browser cookies); no malware that I know of performs social engineering and tricks you into giving up your SSH passwords.
[1] https://hachyderm.io/@damienmiller/117563026759098135
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
If you think about it that’s the whole idea of neural nets
We aren’t smart enough to design the function we want by hand as programmers so we need to learn it through weights
This lesson just keeps recurring in many different ways
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
If I had a free afternoon, I'd ask a bunch of agents to track the people responsible and their other contributions (but usually the scheme is "contribute valuable patches to build reputation, and only inject an exploit when safe to do so").
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
Never enabled sshd there... except on my PostmarketOS PineBook Pro... oh shit!
Is it mainly that it is too big?
I feel like everybody still loves the unix philosophy, but the previous tools were too bad, systemd just filled user demand for easier tools.
The moment someone makes a pid 1 software that is good and still modular, open and pluggable I think that will be an instant success.
Check out dinit, it's basically the same declarative philosophy as systemd, but without the bloat, and with a much simpler config API. Or s6 if you specifically care about "modular", though unless you're an embedded dev, you probably don't actually want that.
But systemd is not a technical problem, it's Redhat (and IBM) having too much control over what happens to the init system.
And of course there is Pottering and his vile Microsoft stint, then another vile stint to enforce age verification on Linux.
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
1. https://news.ycombinator.com/item?id=50383006
https://max.levch.in/post/724289457144070144/shamir-secret-s...
[0] https://en.wikipedia.org/wiki/Eternal_September
This is just negligent. Not a fan of rewrite in rust, but I find it highly unlikely this would happen in that community.
No idea if it's likely to happen though.
So AFAIU, you are fine.
OpenBSD probably did (which doesn't use musl), RedHat probably did (or some other distro using glibc), but they all don't use musl.
[0] https://github.com/leonjza/sshhole-lab
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
[1] https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc121f70f...
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
> - sshd compiled with musl
So alpine as well as a bunch of other embedded Linuxes. Huge install base.
> - not using PAM
What you get if you try to have a minimal setup. Default sshd config if you install from source.
> - password logins enabled
The default. Super common.
> - at least one account with '*' as the password "hash"
Almost every Linux system.
> - at least one OTHER account with a valid password
Almost every Linux system.
So sure, there are caveats, but they cover a massive install base.
This is a super bad bug
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
On top of the embedded linux blindspot, this statement seems unfounded.
> if one IS running sshd, how often is it with unchanged defaults?
By definition, I would imagine the answer to this is most of the time.
https://openwrt.org/docs/guide-user/services/ssh/transfer_dr...
I haven't thought it through, but I suspect in an ideal world, sshd shouldn't have accepted the user password regardless of the outcome of crypt() if the user did not exist and it only pretended to go through the motions to avoid a timing attack.
Did look into their patch yesterday(?) which seemed to mitigate this issue which they (rightfully IMHO) consider caused by musl. But it seems their fix is to do a sanity check on the supposed hashed password char length, and thus still theoretically still allowing the possibility of those "!!!INVALID_PASSWORD!!!" suggestions to fall through....
> Almost every Linux system.
But not, notably, on Alpine, which uses '!' instead.
Edit: or maybe yes on Alpine? I checked my desktop and it has zero *s in shadow, but Alpine's post seems to say they do use * sometimes
From man 5 shadow:
> A password field which starts with an exclamation mark means that the password is locked.
> If the password field contains some string that is not a valid result of crypt(3), for instance ! or *, the user will not be able to use a unix password to log in (but the user may log in the system by other means).
- blue t-shirts and green socks have been added to the default loadout for new hackers, since it's optimal according to AI and costs nothing to the attacker in terms of compute
- there are so many ley line "maps" out there that basically any server placement counts as aligned with at least two, so this one's free also
Disabling sshd password logins is security 101, and the default for some cloud provided images these days. If password logins haven't been disabled, what other doors have been left open anyway?
> The default. Super common.
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
It might actually a good idea, but I would likely avoid it since my built-in Bayesian filter would just mark it as spam.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
Sure there are native Alpine installs too, but that is far less common of a use case.
You'd think they'd all do the smart thing and exec a shell into the container's namespace. You'd be wrong.
OpenWRT also doesn't use PAM by default apparently, and there is a risk of people using password login, which you should never do past initial setup.
But outside specific niches, musl is rare, with good reason. Performance is a joke compared to glibc, especially the allocator. The only reason to choose musl in my opinion is for memory constrained devices (OpenWRT etc) or for building static binaries where performance doesn't matter but portability does. I consider the "small container" use case questionable: performance will suffer, and you could share large base layers instead which would amortize the cost over many different images.
Had to quickly scan the rest of the homepage of HN to confirm.
You had me there :D
LLMs will find it.
They work endlessly at finding the 'and' scenarios.
This should have been closed up a long time ago.
What? Pretty sure Cloudflare detected widespread attacks almost immediately after it was published. We are probably lucky that not much more happened because of responsible disclosure and the large media attention led to widespread admin awareness.
The good news is that considering Alpine is open-source, you can be the one to make the judgement call that it is worth fixing and go out there and fix it ;)
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Edit: You may want to read https://sshhole.com/#exposure.
At this point I feel embarrassed in front of the AIs.
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
"tl;dr" was a bit of a downgrade though, but that's fine
it's a nice way to promote the company behind it
You can use \ to escape formatting.
never changed i guess
We need to re-think EVERY assumption about security.
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
[0] https://news.ycombinator.com/item?id=50377221
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
- I found a way to get AI to help me while I made this
- I asked AI to make this
The people you're objecting to are complaining about the latter, this is an example of the former.
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
Really: from the XZ backdoor in SSH (which went live on some distros) to this...
What's you rationale for calling port-knocking "security through obscurity" and not "defense in depth"?
Put it this way: if I advertize that my server is using port-knocking, that the sequence changes every 30 seconds (like TOTP) and that as soon one sequence is used, it's unusable (so you've only got one attempt to SSH in every 30 seconds, even if you know the sequence)...
Where's the security through obscurity?
But really: even if had musl and PasswordAuthentication allowed (I don't have neither), how would an attacker gets through the port knocking?
More fundamentally the question is: in which way is your system that does not use port knocking more secure than the one who do?
(remember: we're talking about remote exploits by simply having exposed TCP ports open or SSH listening)
Wanna have a talk too about the "solutions" proposed by "security experts" here "because SSH is too old" and that happen to be crufts on top of SSH that had remote root exploitable backdoors? (sorry, holes)
good fucking god. if I get this correctly... you can insta-own any aws/azure instance that was set up with alpine as long as it has ssh enabled, which it probably has.
alpine stocks are crashing?
musl stocks are crashing?
glibc stonks are up
alternative implementations in general slightly down
- It affects Alpine's openssh-server (built without PAM), or openssh-server-pam with UsePAM no, when PasswordAuthentication is enabled. Alpine says every release they checked is affected, 3.16 through 3.24 and edge.
- Fixed openssh packages are out for all of those, including the older 3.16-3.20 releases. Upgrade and restart sshd (rc-service sshd restart). If you can't upgrade yet, set PasswordAuthentication no.
- To check a box: apk info -v openssh-server for the version, and sshd -T | grep -iE 'usepam|passwordauthentication' for the effective config.
- Containers built FROM alpine that run sshd are easy to miss in an inventory. So are the official tiny-cloud images (3.18.11 through 3.24.3), which ship an alpine user with no password and passwordless doas, so a bypass there means root.
- After patching, look in /var/log/messages for sshd lines saying "Accepted password for" an account that has no password.
Anticapitalism.