Jul 8, 2009
Hardware Accelerated Effects in Swing with JXLayer and Decora
Update (August, 31st): I released version 0.2 of this library, fixing a couple of problems, and adding the possibility of making your own custom pixel shaders. You can find the second part of this series on pixel shaders here.
The Olive Branch
A lot has been said about Swing and JavaFX recently, in a seemingly unending flame war between fans of each side. Both have merits and flaws, and as you’ve seen over time, i’m advocating a 3rd way, which i decided to pompously call The Olive Branch for no reason whatsoever, which is using the best of both worlds.
The plan is to use, directly from java, scenario and decora -the graphics libraries that power the javafx runtime, and which i love probably more than my girlfriend, but don’t tell her (: – (btw if you don’t like crappy jokes, you better stop reading now, and grab a taco or something) . Of course, things have changed since their release, what was once an open source public API (albeit destined to change heavily) is now an internal proprietary one, which means danger, will robinson. So, the plan is to have the community build things that are so cool, and innovative, that Sun will allow us to do just that and make scenario/decora public and usable from java, as they said they would, like with a jnlp extension or something. I think when things have stabilized a little, they’ll get to it. But at the moment, i have succeeded in doing that, at a level of 2% (and i’ve mailed Sun telling them everything i’ve done with scenario in the last 18 months – which is quite a bit in retrospect). Let’s see if we can crank that baby up.
Introducing jfxlayer
I’ve started this a long time ago, i wanted to help people use scenario and decora in swing, not only in a pure scenario scene as with Scenile, but from regular, real world swing, because some tasks are better handled by scenario and decora, and vice versa. The best of both worlds. I never got around to releasing it because i haven’t gotten to a satisfying point in this endeavour (+ i’m lazy), i’ve just planned the whole shebang, and coded the first little step, how to use hardware accelerated decora effects in plain old swing. It’s like 30 lines of code, that i kept in sync with every JavaFX release. I’ll get to the other planned features eventually, in the meantime, here’s the result of this piece of work. Hopefully, it’s interesting enough for regular Swing java users, or more “exotic” ones, like them groovy folks (:
It’s almost nothing code and work-wise. However, this opens a *huge* realm of possibilities for swing apps, and i hope you’ll see that too.
The concept is simple: add decora’s effects to the BufferedImageOps effects that jxlayer allows.
Decora. OpenJFX. JXLayer. JFXLayer.
Decora ?
Decora is Chris Campbell’s baby, if you don’t know who that is, you can stop reading now, bye. This library is arguably the smallest, the easiest to use and most powerful one you’ll ever come across if you’re dealing with effects, image processing, and shaders. A decora effect is made with a JSL file (Java Shader Language) which is basically a mix between Java (in order to get/set parameters and control the effect) and GLSL/HLSL ie, a generic high level OpenGL/DirectX shader language. This JSL file is then “compiled” (by a code generation utility) into a specific effect for each supported backend. The backends are mutually exclusive branches of code that target a specific architecture/3D API, etc. Depending on your software and hardware configuration, decora will choose the backend that will perform the best on your computer, whether you’re on windows mac or linux for instance. You have a regular “software” backend in pure java, one using JNI, one targetting OpenGL, another one OpenGL ES2 for mobile devices, another one Direct3D, and one targeting Prism, the next gen JavaFX graphics renderer, which will allow 3D and more perf, which is about as far as everyone knows. Back in the open source decora days, i’ve also written a backend myself, which multithreaded the regular Java backend, so that effects would use all your available cpus/cores.
All that from one tiny JSL file, they really are small, check the old open source version you’ll see. For the user, it’s of course transparent, you create an instance of the effect you want, and use it, decora handles the rest behind the scenes.
Sure, there aren’t that many effects yet (in fact there haven’t been any new ones since it was released, the work is done on the architecture, perf, backends, for the existing effects, which is very logical if you think about it) compared to, say, what Jerry Huxtable offers in his jhlabs filters library. In practice, it’s enough, for instance you have blurs and shadows, a perspective effect (think coverflow), color transforms, etc and a way to mix and chain all of the effects. If we had access to the compiler (or if we reverse engineered and forward ported that to the open source version) we could add new effects, but it’s not the case anymore. I think it’s doable if you really want it, especially for GLSL shaders which’ll be very interesting once Prism is released, if the compiler isn’t available at that time. I haven’t felt the need to create new effects until this week where i needed a better displacement effect than the one provided. So we’ll see how it goes but i’m not planning on doing it anytime soon.
How do i use it in swing ?
Enough about decora. In your Swing app, use JXLayer, and use the JFXLayer effect bridge with stock decora effects. It’s very simple:
BufferedLayerUI ui = ...; // your regular JXLayer LayerUI Effect decoraEffect = ...; // your regular decora effect // like GaussianBlur, Bloom, DropShadow, etc ui.setLayerEffects (new DecoraLayerEffect (effect));
Compare that with using JOGL and pixel shaders directly as Romain did in the Filthy Rich Clients book (the sample is like 500 lines long, and which are mostly low level code for setting everything up for rendering, here it takes 5 at most, thanks Chris!)
Demos. Finally! I’m as tired reading this awfully long blog post as you are writing it
I know you’re all about demos and eye candy, and today, i have 3 of them for you. Not one, but three, and if you call right now with your credit card number, i’ll add a set of Ginsu2000 knifes for only 3 easy payments of 9.99$, they can cut a shoe, you know ?
Since this post is about hardware accelerated effects, you’ll obviously need a decent graphics card with up to date drivers.
The first one is pretty basic, it’s two buttons. Pushing each one starts an animation (using the same code that powers the eMotionBlur demo of last time) that progressively blurs the component (but remains live). The button on the left is using a jhlabs blur filter, while the one on the right is using decora’s one. At the default size, it’s really close, but maximize the window and you’ll see the difference, and possibly start to envision the possibilities this offers. Who knows maybe it can be used to power screen transitions (even though i’m still not fully satisfied with the perf yet – maybe Alexander Potochkin, or Dmitri Trembovetski can help us out here, pajalusta (or whatever it’s spelled in cyrillic (: – you’ll see it’s decent enough for me to release it) – Mark, you can keep us updated on that (:
The second one is a modified demo from the JXLayer distribution, using the LockableUI, here also you can compare blurs. On my machine it’s almost instantaneous with decora.
And finally the 3rd one is a modified version of Twimber, my tiny twitter provider for Kirill’s great Amber project. I’ve hacked animated blooms, blurs, and color transforms into it in a matter of an hour, where it would have probably taken me days fiddling with jogl. Bear in my mind this demo is *heavy* with animated effects, and absolutely not optimized at all (for instance the repaints are too big, the effects target components that are too big, etc) and might run slowly (it does a little on my machine) or quite possibly crash your vm (which it does on another machine, probably a bug in my graphics card driver that crashes everything), plus there are some deadlocks somewhere between Amber and the latest releases of Trident, that i haven’t tried to remove, maybe Kirill can help us with that, if need be. If that happens, you can only close the webstart window by killing the process, sorry about that, guys. Hopefully, we fixed the problem, it stll won’t be optimized though ^^
Is it safe to use ?
Sure, using internal APIs is going to be a pain in the neck, it already has been for me. But, all in all, Decora’s API did not change much in the last 18 months, and especially not the effects’. And as the saying goes, no pain no gain. Plus they work fine with JavaFX 1.2 and JFXLayer right now, you don’t have to use the next version if you don’t want to have to modify them.
Downloads and source
Update (August, 31st): As mentioned earlier, I released version 0.2 of this library, fixing a couple of problems, and adding the possibility of making your own custom pixel shaders. You can find the second part of this series on pixel shaders here. I didn’t change the code linked here for archive purposes though.
You can find JFXLayer here (zipped source project + demos), and twimber2 here (zipped source project).
Let me know what you think, if the demos run well, or whatever in the comments, or sooner on twitter.
Do these demos work as well on all platforms? Any chance of showing the FPS for each pane in the first demo?
I was also wondering if Sun/Chris have ever talked about firming up the API for Decora for Java/Swing users? In the JavaFX != JavaFX Script sense. I know Scenario changes have caused you some headaches and commonly described as an ‘implementation detail’ i.e. dont rely on it not changing without much warning. What I’m getting at is Decora stable or just-hasnt-changed-yet?
Incidentally, would you write off Scenario for Java users now leaving Trident as the only current/supported animation lib? RIP Scenile?
One final possibly awkward question, where do developers stand on being able to distribute decora/javafx jars with their own products?
I tested those demos under windows and linux, both with fairly recent hardware.
I really can’t tell for Sun, but from my perspective it seems that the focus is FX script for the moment, when that’s more advanced/stable/what have you, you never know. To me it’s pretty obvious people would want to leverage the respective strengths of each “platform” instead of fragmenting efforts and possibly losing investment, know-how, etc.
As for Decora, it has changed a lot internally, (the backends and their respective renderers, + the effects’ peers – the in fine implementation of the effect – as well), and is still changing as we speak (JIRA says it all), but not the effects API which is what you’d use with JXLayer here. The internal changes are only impacting me as it modifies the way you apply an effect to a java2d image, ie the DecoraEffectLayer impl only.
As for Scenile, no it’s not dead or obsolete at all, i have updated it for every release too, but never blogged about it ;) I have been working on some exciting stuff for it, at least to me. It’s the only way i can get a scene out of a visual tool, so it’s pretty much vital to me. Of course now that JavaFX 1.2 has made my life way harder to use it from java, i have more work to do, but i’ll get to it. For the animation part, yes, Trident is pretty much the only animation library that’s alive right now. You could also use Scenario’s 1.1 support.
I’d love to have the distribution clause relaxed a bit, i intended to use the JavaFX runtime with OSGi and that clause prevents it, so let’s cross our fingers.
I hope Sun can give us some information on all those questions.
Actually, I saw no difference between Decora and the other blur effects.
But now, I have rather old hardware (4 or 5 years old…), so perhaps Decora just falls back on computed blur, or something.
I should try on more recent hardware.
I find your experiments interesting, I hope Sun will soon free these libraries.
This is using the stock javafx 1.2. Check the javafx demos, if they’re not hardware accelerated, then those demos won’t be either, obviously :)
Do you know if together with Prism, JavaFX will expose JSL for custom shader programming? That would kill. Chris is really mean to create that sort of stuff and keep it for himself. ;-)
hehe osvaldo, i wish i knew :)
I’ve asked richard bair about it on twitter, but no answer yet.
I would so love to have access to decora’s compiler back, we could have in java, and javafx, the features flash’s pixelbender and silverlight/wpf hlsl effects already offer. Access to the compiler is the only thing that prevents us from doing that, it’s too bad, i know.
You might have a good point, if prism offers better performance and 3d integration/acceleration, that will probably help in stabilizing decora/scenario, and getting to a point where Sun’s happy enough about them that they might let us play with it. We can only hope, unfortunately i have no idea if/when that’s going to happen.
Hehe, Richard probably can’t answer that, I wish JavaFX’s development could leave soon its stage of private development… a big, disgusting step backwards after the last years of Sun moving everything to open source… but probably a necessity, until they “cross the chasm” in adoption and competition with other RIA techs; they are all issuing updates frenetically and trying to kick each others’ butts with first or exclusive features, and some secrecy is part of the game.
On public JSL, I hope the JavaFX leads consider that this may be a serious competitive advantage. The other guys have important advantages. Flash is the incumbent, with ubiquitous deployment and design tools that are first-class and loved by all designers. Microsoft has enormous resources and they’re delivering an impressive volume of new functionality in short iterations; and they already have at least one killer feature, smooth streaming (although this is apparently more a server-side IIS Media Services feature) and I guess the HLSL facility can be great to do very cool stuff in realtime on top of streaming video. Now, JavaFX is still the underdog; while it has some developer-centric advantages like the really great JavaFX Script language, they need user-centric killer features, in other words “shiny stuff”.
This is really good work! However, I’m not sure how to get this to work on my end. Firstly, where should the dll’s go and how do I know when I’ve succeeded with getting this hardware accelerated. I mean, I have a class that renders a Gaussian Blur on a JXLayer, but it is really slow so I guess all calculations are done in software. I don’t know much about the low-level details of swing/java2d/javafx (or java for that matter) so it would be great to get a short description of where to put all this stuff (the dll’s) if one wanted to distribute something more “real”. :)
Great work though!
For deployment the only thing you need in addition to JXLayer is a dependency to the JavaFX runtime, as you can see in the jnlp files for the demos.
The jars/dll are only used during development, for compilation and testing. Get the jars and dlls from the JavaFX SDK, and put them somewhere in your project folder (eg: lib). Add the jars to the classpath, and run your app with the folder where you put the dlls as the library path (in our lib example: -Djava.library.path=lib or something similar).
To know whether an effect is accelerated, take a look at the DecoraLayerEffect and DecoraEffectRenderer to learn how to use the Effect.getAccelType() method, which returns the type of acceleration that is used with this effect.
That should be it.