While the OmniFish team and other GlassFish contributors are intentisvely working on GlassFish 8 (see milestone releases), bringing Jakarta EE 11 and other new features, more general improvements and fixes are happening also in GlassFish 7, where we expect a new minor version soon. David Matejcek, Eclipse GlassFish Project Lead and OmniFish co-founder, walks us though the recent work on the upcoming GlassFish 7.1.0 version and what’s coming.
Question: Welcome, David. Can you start by telling us about the major updates in the upcoming GlassFish 7.1.0 version?
David: Hi. We’ve implemented quite a few changes in GlassFish that will be released in GlassFish 7.1.0. While some are technically breaking changes, they’re not major. Applications built for GlassFish 7.0 should generally still work on 7.1. However, one key difference is that 7.1 requires at least Java 17, and specifically, it won’t work with Java 11.
Question: Could you elaborate on those breaking changes?
David: Certainly. One example is a problematic order of command-line arguments in the asadmin tool. It’s still a common mistake users make when using arguments in the deprecated order, and they will need to adjust their scripts. This order has been clearly documented as deprecated for about 15 years, both within the asadmin tool documentation itself and in the documentation of each specific command.
Besides that, raising the requirement to run GlassFish at least with Java 17 is itself a big breaking change for those, who still run it on Java 11 to 16. We did it to allow us upgrading to latest versions of dependencies that require Java 17+, and integrate MicroProfile APIs to all GlassFish distributions, including GlassFish Web Profile and Embedded GlassFish. Speaking of MicroProfile, GlassFish 7.1 is going to support MicroProfile Health, which will be a completely new feature.
MicroProfile APIs and the MicroProfile Health addition
Question: Interesting! MicroProfile Health will be a new feature in GlassFish 7.1?
Yes. We were thinking about adding MicroProfile Health for a long time, because it’s a popular feature in other servers, very useful in Kubernetes and cloud deployments. And it’s also pretty simple and straightforward. Then Thiago join our team and took the task to implement it. We knew Thiago from his previous brilliant contributions in the Piranha Cloud project. And he did a very good job with MicroProfile Health in GlassFish too.
MicroProfile Health in GlassFish is a completely new implementation, written from scratch. We considered reusing some existing implementation like SmallRye or Helidon, but their implementations would bring a lot of unnecessary dependencies. MicroProfile Health is a pretty small specification and so we decided to implement it from scratch. We finished the implementation pretty quickly, almost half a year ago. However, we decided to base it on Java 17 because most of the other MicroProfile components in GlassFish require it too. And then we needed to wait until we release GlassFish 7.1, which will also require Java 17.
Question: What about other MicroProfile APIs? Weren’t some of them already supported by GlassFish 7.0 even on Java 11?
Actually, they were supported only on Java 17 or newer. Although GlassFish 7.0 provides MicroProfile Config, Rest Client and JWT, they are only available when you run GlassFish on Java 17+. GlassFish server loads modules dynamically, and we tuned GlassFish to recognize the Java version and load MicroProfile modules only on the required Java version. And this is only in the full GlassFish Jakarta EE Platform edition. GlassFish Web Profile edition and Embedded GlassFish edition don’t include MicroProfile modules. Raising the requirement to Java 17 will allow us adding all the exising MicroProfile modules to all Glassfish editions, including Embedded GlassFish.
Keystore Migration and TCK Challenges
Question: Besides the Java 17+ requirement, what will be other significant breaking changes in GlassFish 7.1.0?
David: Another significant change is the shift from the JKS (Java Keystore) certificate format to PKCS12. PKCS12 is the industry standard, and while GlassFish didn’t previously fully support it, we’ve now made the transition. This change was quite a bit of work. It took us a couple of days, as it touches many parts of the system. There are a lot of hardcoded instances of the old format, especially in the Jakarta EE TCK (Technology Compatibility Kit) configuration, which we had to patch too. While it was manageable, it was a substantial effort to update everything.
Question: Speaking of the Jakarta EE TCK, you’ve been working on a new approach for it. Can you tell us more about that?
David: Ah, yes, the Jakarta EE TCK. The TCK tests used to take around 8 hours to run and relied on old Ant scripts and an extremely fragile configuration. Now, the Jakarta EE 11 TCK tests utilize Arquillian. Each specification project has its own team maintaining their specific setup, and they all have their own evolution. The point is, now when we run the TCK tests in CI, we have a much simpler Maven configuration and executed with A simple command, without needing any 20-headed monster to configure and run the TCK. It’s also easy to adjust the configuration, e.g. to run against a different GlassFish version, using system properties and simple changes in XML files.
Question: That sounds like a significant improvement in efficiency. Are there any remaining issues with the TCK?
David: There are still a few things that cause issues, primarily because the way how we run the tests relies on the full Jakarta Platform TCK distribution . This is a 0.5 GB ZIP file, which, at the end of the day, is quite large and has to be downloaded before each run. And sometimes the download fails, causing the whole test suite to fail. So, while we’ve made progress, there’s still work to do to get all tests passing reliably. My team at OmniFish is currently focusing on fixing those remaining issues.
Upgrade Now: GlassFish 7.1 will require Java 17+. Ready to take the leap? Have questions, need help and experts on your side? Book a free consultancy call with us or Send us a message.
JPMS and modularization
Question: You’re also exploring the use of JPMS (Java Platform Module System). What are the benefits of moving towards JPMS in GlassFish?
David: Java JPMS modules are very strict about how the code and modules are structured and assembled. Adopting JPMS in GlassFish makes it necessary to tidy up the code, otherwise the code won’t compile or will raise exceptions during tests. JPMS helps us identify issues and address many old problems, including security vulnerabilities that might have existed or could arise, and that we might not even be aware of yet. As we embark on this path, issues gradually surface. For example, during our migration to Java 17 and JPMS, we faced significant challenges with reflection and proxy generation. Java really dislikes it when you generate a proxy for a class in a completely different package. So, we had to change the packages of generated EJB proxies to respect Java’s rules, even when GlassFish wasn’t fully on JPMS yet. And there are yet many places where it still needs to be fixed to meet the strict requirements of JPMS.
Question: So, it’s forcing a cleaner, more secure architecture?
David: Exactly. We still have some add-opens and exports in GlassFish, but our goal is to reduce those. For example, using add-opens for “all unnamed” modules, which we still do, is not ideal from a security perspective. It means that if a user deploys an application, internal functionality of these modules, for example javax.naming, is open to reflection for code in the application, which is a security risk if that application is compromised.
Our idea is to create a specific module that’s strictly controlled by JPMS, where we can place anything that Java’s public functionality doesn’t allow. Then, through a targeted add-opens, we can explicitly grant permission only to that specific module. This ensures that only our internal code can access these functionalities, keeping GlassFish, Java, and the user secure from external interference.
Addressing Race Conditions and Testing Philosophy
Question: You’ve also been tackling a lot of race conditions. Can you tell us about some of the challenges and solutions you’ve implemented in GlassFish?
David: We maintain a huge test suite for GlassFish. The Jakarta EE TCK is only a part of the test suite. And we’re adding more and more tests to ensure all works well. And then suddenly some tests start randomly failing, just once in a while, even though there was no change in GlassFish that could introduce it. And sometimes all tests start failing, because of some infrastructure issues, like recently when GitHub CI couldn’t download a recent Maven version. But that’s another story.
Race conditions are tricky. For example, I found a bug where calling interrupt() on a thread didn’t always immediately stop it. I always thought interrupt() meant the thread would throw an exception and be done, but it can actually catch the exception and continue running. So, we had to add a join() to ensure the thread truly finished after it was interrupted. What’s even more insidious is that interrupt() is an asynchronous event. The thread might only throw the InterruptedException when it calls a blocking operation or actively checks its interrupted status.
Our tests were randomly failing in GitHub CI, perhaps one in a hundred times, and it was getting worse. The frustrating part was that we couldn’t reproduce it locally when we added logging. With more logging, the thread would block waiting for output, and then the InterruptedException would fire, and everything would work as expected. But when I removed the logs, on some machines, with fast file systems, without any blocking, the exception wouldn’t get thrown, and the thread would run indefinitely.
Question: So, the logging itself was changing the behavior and masking the race condition?
David Matejcek: Precisely. It’s a subtle trap in Java. Many developers will put a Thread.sleep() and then just catch InterruptedException, print an error, and continue. But when you catch InterruptedException, the thread’s interrupted status is cleared. You need to re-interrupt the thread if you want the interruption to propagate through your code. And then you have to actively check for the interrupted status if you don’t have any blocking calls that would throw an exception. It’s about being aware of the causality. You can’t rely on latency. The system must always behave deterministically.
My approach to these issues is to use JUnit repeated tests. If I suspect an InterruptedException or a race condition, I’ll run the test thousands of times. A few milliseconds for a thousand runs is nothing compared to the hours you’d spend trying to debug an intermittent failure. If a test fails 9 times out of a thousand executions, even if it worked fine for all the executions on another machine, it indicates a problem that needs to be fixed.
Other Improvements and Open Source Contributions
Question: You mentioned GitHub Actions and infrastructure issues failing automated tests. What happened there?
David: Interestingly, it wasn’t GitHub Actions directly, but the Apache Foundation download server was down, possibly due to maintenance. One night, our tests started failing because we were trying to download Maven 3.9.9, and it suddenly resulted in connection timeouts. On my machine, all worked, only tests executed in GitHub were failing. Then I realized that GitHub Actions continuously updates their CI containers. So, while we had specified Ubuntu 24.04, the underlying software on those containers was being updated in the background. It turned out they had already included Maven 3.9.9, so I simply removed our explicit download, and tests started working again.
Question: You’ve also been busy refactoring and making other improvements. Any highlights?
David: Yes, I’ve been doing a lot of cleanup. One thing that really frustrated me was seeing System.setProperty and System.getProperty calls scattered throughout the code without proper management. When you update dependencies, these random property changes can cause builds to behave erratically, and it’s a nightmare to debug. So, I’ve been centralizing and cleaning those up.
Also, we reduced the usage of the keytool command line tool in GlassFish and replaced it with the Java standard API to manipulate key stores. We still need to use keytool in some cases because Java itself doesn’t have a standard way to generate self-signed certificates. We’d have to either use Bouncy Castle or resort to reflection on internal Java classes, which is very brittle as those classes change across Java versions. Generating a self-signed certificate is crucial for domain creation, as we can’t start the server without one. I strongly advise users against using domain1 with the default certificate and instead to create a new domain with custom settings tailored to their environment.
Question: It sounds like you’re making significant strides in improving the reliability and maintainability of GlassFish.
David: Absolutely. We’ve tackled more race conditions in distributed transactions that we’ve known about for years. We’re trying to be aggressive in addressing these to make sure that GlassFish is reliable even in situations that we can’t easily predict. The more we do this, the more experienced and efficient we are getting at it.
Question: What about the ongoing development and community contributions?
David: We’re always reviewing pull requests timely. Piotrek, a fellow committer on GlassFish, has been doing some great work. He’s an open-source contributor who works as a freelancer, and he’s actively releasing fixes.
We’ve also made decisions like dropping support for GlassFish 6.2.5 in a new version of the GlassFish Arquillian container maintained by OmniFish. Users using such an old version of GlassFish should stick to older Arquillian versions. For GlassFish 7 and beyond, we’ll be introducing new features. For instance, in Arquillian, we’ve fixed bugs caused by previous fixes, and I’ve added a crucial log that shows when an application is deployed to GlassFish. This helped me find a bug that was breaking TCK tests, where Arquillian was failing to deploy properly without reporting it, while deploying the same application by standard means worked.
Enhancing Logging and Security Measures
Question: You mentioned that improving stability has been a significant focus. What improvements have you made to GlassFish’s logging capabilities, for example?
David: Logging is often the culprit, or, at best, a silent observer of many system failures. The challenge with logging, especially in Java, is that it can be incredibly slow if not optimized. This led to the proliferation of various logging frameworks like Log4j, Logback, and SLF4J, which were faster and more configurable than the default Java API. However, the recent Log4j security vulnerability highlighted the risks of external logging libraries.
In GlassFish, while we have some reported vulnerabilities, they mostly concern the administrative interface and not low-level functionality like logging . My biggest piece of advice for people running GlassFish in production is never expose the administrative interface to the internet. Even with encryption, it’s best to use a VPN or, even better, place it behind a strictly controlled, authorized access layer that logs every login attempt, origin, and action. While GlassFish itself logs actions, it’s ideally a network application firewall like Cloudflare or Nginx that should handle this at the perimeter, validating requests and blocking suspicious activity before it even reaches the application server.
Question: That brings up a good point about security best practices. Beyond securing the admin console, what other security-related changes are you implementing?
David: We’re also addressing internal practices that could lead to security vulnerabilities. For example, I’ve worked to centralize and control how system properties are set within GlassFish. Previously, System.setProperty calls were scattered throughout the codebase, making it hard to track what was being set, by whom, and when. Now, when GlassFish sets a system property, it will be logged. This helps us track down rogue property changes, like issues we found where JMX was unexpectedly modifying the KeyStore settings, which should never happen.
We’ve also found and fixed unclosed sockets and other concurrency issues that were causing intermittent test failures. These are often difficult to reproduce because they depend on timing and system load.
Trust Stores and Certificate Management Challenges
Moderator: Let’s delve deeper into certificates and TrustStores. What are some of the complexities there, and how are you addressing them?
David Matejcek: We’re making progress with certificate handling. We modified some tests to explicitly create a new certificate file instead of importing GlassFish’s default one. The approach to certificate management varies from test to test, but we’re moving towards a standardized method.
We also encountered situations where trust stores shouldn’t be loaded from the system but from a central repository. GlassFish, with its own trust store, isn’t fully compatible with this yet. Ideally, the operating system should handle this.
One option is to keep a separate trust store in GlassFish and provide admin commands to help users import certificates. KeyTool is pretty good with PKCS12 now, which is great. The challenge is that these tools operate externally. You prepare the key store file, and then you swap it out at restart. I wouldn’t do this during runtime because there’s no way to automatically signal the application to reload it. You’d also introduce a risk of security vulnerabilities if an external process could manipulate the key store while the application is running.
Moderator: So, you’re advocating for a more controlled, pre-startup approach for certificate updates?
David Matejcek: Exactly. While some users might want to synchronize a central file, I consider that a security incident if it’s done at runtime. A key store should be locked down. But there are scenarios where a central, well-managed file of secure certificates, maintained by a dedicated team, is more secure than having individual keystore files across multiple GlassFish installations. It’s about accountability. We need to guide users towards secure practices, not enable potentially harmful behaviors like merging of certificate stores from random places. That would lead to hours of costly consultations trying to debug issues!
It’s a complex problem with no simple solution, and it definitely needs proper investment. For now, it works well as it is, although it might seem counterintuitive and too restrictive during development.
Concluding Thoughts
Moderator: Do you have any final thoughts or advice?
David Matejcek: We’re always trying to make GlassFish more robust and user-friendly. For instance, the logging now includes timestamps with microsecond precision in GlassFish 7. This might seem small, but it was incredibly frustrating when dealing with fast machines where multiple log entries would have the exact same millisecond timestamp.
My general advice, especially for developers, is to understand why something is done, not just what. Comments like “here I set x” are useless if they don’t explain the reason. If your code is too cryptic, make it a well-named method instead. Clarity is paramount.
Moderator: That’s excellent advice. Thank you for this insightful discussion, David.
David Matejcek: My pleasure!
