Lqd's Journal

A great man once said: “I love it when a plan comes together”.

It’s time to explain the final pieces in this little pet project of mine, and release the whole as a (already rather usable) preview.

Swing has many advantages, let’s agree on that, but on the other hand it has many, so many, things that just make me want to jump out my 1st floor window when I use it. None of those are deal-breakers per se, but it does feel a lot like ‘death by a thousand paper cuts’. This is my attempt at correcting those pain points, and making using Swing a bit easier.

The problems I’m trying to solve

Without a doubt, the biggest pain point in using Swing is its API. First of all, components have a huge number of methods, it’s really frightening. A lot of them are useless AWT methods. To me, the most important qualities of a good API are related to the user experience: conciseness, and discoverability.

As a user, I want to write short amounts of code, and I don’t want to need to remember a lot about how the API is used, or read a ton of docs to do simple things. Swing fails on both of these points.

I could go on. The whole event system requires a lot of classes and methods. There’s no way to group components in a logical group, only in a physical group. So if you want to say hide 5 components, your choices are 5 setVisible() method calls, or group the components in a panel, and hide that one instead. The latter offers a nicer way to hide the components in one call, but comes with performance and layout problems related to adding another component in your UI.

And so on.

jQuery’s mantra is ‘Write Less, Do more‘, and that’s a motto I can get behind, no matter the programming language. How does jQuery achieve clean and concise code, and how would the concepts it’s built on look in Java, and Swing ?

The answer to that has multiple parts, and by now you’ve seen most of them. A query engine (hQuery + Shizzle) will help you select one or more components in your UI – which allows you, if you want, to not have to create a member variable for each of your components. You’ll then interact with the results of your query using a composite – reducing boilerplate code – or your member variable components as usual, using Swing wrappers/builders (HTk) implementing a hopefully easier API. Let’s start from the top of the stack and build our way down.

HTk

There’s multiple ways to make the Swing API easier to use. I could modify the Swing classes directly, but that would require too much work. I could make new Swing components performing the functions of existing ones, but that would force me to extend JComponent at the very least, and having a lot of methods I don’t want in my API. The last of the most obvious methods is I could have builder or wrapper classes that create or configure an existing JComponent instance, and only interact with the builder.

The latter allows you to start with a clean slate, and offers the best work/reward ratio.

HTk is a small set of builders for the most common components and methods I use (the version I’ll release today only targets the most basic components, but the version I use contains builder for more advanced components like tables and list. I want to wait to have more experience using and improving them before releasing those). I already talked a very tiny bit about them last time when we talked about Shizzle, and today I’ll describe them in more detail.

For “each” of the JComponent classes, there would be a HComponent class that would provide the builder for the associated JComponent. HButton builds a JButton, HLabel builds a JLabel, and so on. The code still looks like you’re using Swing components, but the HComponents are never added to the UI – In turn, it also creates a problem, since you don’t have any HComponent in your panels, when you want to query them (say, disable all the buttons that apply to a table row when there’s no row selected in the table, this should be one line of code, and now it can be), so each HComponent adds style classes so you can still query them, as we’ll describe more in the next section.

So when you create an HComponent, it’ll create a delegate JComponent to configure, or you can pass it an existing JComponent instance – since you might want to interact with the components using the same API regardless of whether you’ve already added them to your UI or not. This delegate JComponent can be accessed with the asComponent() method. This is useful to know because as I’ve said the point is to reduce the number of methods in components to most useful ones, and you need to be able to access the underlying JComponent to call the less common ones. HTk is not a new toolkit whose goal is to replace Swing, the goal is to help you for the most common cases you’ll encounter (right now it’s of course biased towards the most common cases me and the friends at my company encounter – in our own projects and the client projects we use it in; but releasing it will hopefully start a discussion on what your needs are, dear reader).

The other methods are either just getters/setters and methods from the underlying wrapped JComponent class, or new methods to simplify the existing API (eg, events) or new features such (eg, style classes).

For conciseness, the HComponents support method chaining and I wanted to talk a bit about that. Since setters usually return void, and chaining setters actually looks weird – see component.setX (2).setY (0) – I started having with* methods too, a fairly traditional way of having a setter returning this. However, that made having 3 methods for each property instead of just 2, which went directly against the design principle of limiting the number of methods. So I decided not to use the JavaBeans notation and I used the name of the property for the getter and the chainable setter, mimicking named arguments in a sense – component.x (2).y (0). As I don’t need to use reflection on those components, it has not created any problems yet. Nothing is set in stone, we can talk about it, maybe you have a better solution.

Since the goal was just to wrap Swing objects, just to have a smaller API, from blank or existing instances, I chose to have stateless components, and then little by little we thought we could add some features that required state (say a suffix in a JLabel), and then also wanted to create custom components. I don’t have the complete answer on this yet, things can be improved greatly to make it easier to add features and create custom components. Until now, the focus has mostly been put on using existing Swing APIs, rather than creating new components, so it’s understandable – How this is done right now, is the component’s state, when it needs one, is stored using client properties in the delegate JComponent. It’s a little bit tedious (for people who write HComponents, so you’re safe here) and we’ll talk about this later in the ‘What can be improved’ section.

The more common new methods in HComponent are used for queries. Naming a component with as (“aName”), and handling its style classes (addStyle and removeStyle, hasStyle, toggleStyle, etc). Then there’s also a number (actually too big a number) of methods dealing with the component’s border some dealing with sizes, bounds and location, property change listeners, etc.

Since I just mentioned property change listeners, I’d like to take a moment to describe what the plan is for events. Still a work-in-progress, but I added some example code in this preview, and I think we can do the same thing other toolkits have done, and just use one method call to add a listener, and one to remove it, whether it’s a regular Swing event or a new one (for when custom component creation is more fully fleshed out). You have a generic EventSource interface that binds the event type to its listener type (say ActionListener to ActionEvent), which would be used when registering listeners. HComponents have a connect() method, which takes an EventSource and an event listener. You’d then have an EventSource object for Swing events (in the preview code you can see ActionEvents, MouseEvents, MouseMotionEvents, MouseWheelEvents) and another class for custom event sources (based on composites of course, which would then use another generic listener interface EventListener). For Swing listeners, it’d look like this: component.connect (Events.mouse, aRegularAWTMouseListener) or component.connect (MouseEvents.click, aRegularAWTMouseListener). I guess you could also have component.click.connect (listener) if you wanted to. The HComponent will then call the appropriate addListenerType method.

Enough about components – and specifics can be seen in the code – and let’s turn to the container, the HPanel class.

At its core it’s very basic, providing standard methods to add and remove JComponents or HComponents, or setting the layout manager, etc. But there’s really two things of interest here. The first is the logical grouping. I’ve mentioned style classes before, and they’re part of the answer to the lack of logical groups (querying for components matching a set of rules in general, and most of the time having a specific style class, allows for an easy way to get a group of components regardless of where they are in the component tree). The HPanel offers a way to automatically add a style class to a whole bunch of components without adding the style class to each of them manually, it’s a little helper that can be useful. The group method returns a StyleGroup object which’ll just have add() methods. By calling them you’ll add your component to the panel as usual, and also at the same time the group’s style class will be added to the component’s existing style classes. Like so panel.group (“myStyle”).add (component1).add (component2). I decided not to use varargs because you can add both JComponents and HComponents to an HPanel and thus to a StyleGroup, and so you couldn’t have one method that’d take a variable set of JComponents and HComponents.

The second thing of interest is the most novel one, it’s the link with the next layer, the query engine. So there’s methods taking a String selector query and returning the handle to the result, an HQuery instance.

hQuery

Now that HTk provides a smaller API over Swing, we can follow jQuery’s path and provide an object that would help with conciseness over multiple similar method calls, or as a way to remove the need for having a member variable for every component in your screen/page (as the default Netbeans/Matisse settings generate).

A long time ago we saw that composites allow for concise code, we still need a way to fill the composite with values, to loop over and do its proxy job. Last time we saw Shizzle, a query engine that worked on Java and Swing objects using the CSS selector syntax, this will give us our components. This is where hQuery comes into play, as the glue code between a query and the results’ composite.

Again, hQuery is ridiculously small, you only need to know about 1 class (actually 1 interface; there’s also two implementations, both of which you’ll never need unless you want to extend the library. As I said very small).

When you call the query method on a panel, it’ll return an instance of the HQuery interface so you can interact with the results. Its implementation uses Shizzle, to query the panel component tree, and fills a composite with the results, really simple. I like to create a HQuery $ (String selector) { return query (selector);} method on my page/screen class (extending HPanel) so I can also have a short way of calling it.

So now $ (“aQuery”) returns an HQuery instance, but what can be done with it ? Mostly three things, the fist one being calling some methods defined at the HComponent level. For instance, returning to our early example, if I have a table and a set of buttons launching different actions, some of which only apply to the selected row (for those I would have just used a StyleGroup or individual addStyle calls on those specific buttons). In my selection changed listener, all I’d need to do is call $ (“.buttonsNeedingSelection”).enabled (false) if the selection was empty and all those buttons would be disabled.
Similarly, you can control the visibility of the results, manage the style classes, add the components to another panel, or detach them from their current parent container, or connect/disconnect an event listener, etc.

The second things an HQuery instance offers is to do some other query, using the results as a starting point. For instance you could use find() and search for some of the results’ descendants, or use the parents() methods, this time to query the component tree upwards.

And lastly, but the most useful in practice is getting some kind of handle on the results (a composite most of the time). Actually all the first set of methods, including the $ (“.style”).enabled (false) example we saw, create a composite on which they call a HComponent method, enabled() in this case.

As there’s different ways to use the results, there is going to be different methods depending on whether you want to loop over the results yourself or use a composite. In the first case you could use contents() to get the List of JComponents, but there’s another way. asJComponent() (if you have only one result) and asJComponents() (if you have 1 or more results) both take a class that extends JComponent, so you could do for (JButton b : $ (“JButton”).asJComponents (JButton.class)) {}. The same method exists for HComponents too, and since we’re querying inside a live UI containing only JComponents, every JComponent will be wrapped by an instance of the HComponent class you specifiy.

So these methods give you a List, but this is not the way I prefer to use hQuery. I usually want a composite of the results, and these are the last of the important methods this object provides. The asComposite() method takes a Class extending HComponent, and obviously returns a composite of the results accessible with a proxy of the class you specified (the parameter-less asComposite() method returns an HComponent composite, which is the most basic one, the ones used by the HComponent methods at the HQuery level). And there’s also 2 similar methods for composites on JComponents.

Most of the time I use the as() method, which just calls asComposite(). I’ll show you an example just after I describe one last trick. Since I’m using quite a similar API, I decided to apply Fabrizio Giudici’s cute class literal trick and I added in each of the HComponent classes two public Class variables for the component itself and the JComponent class it wraps. With a simple static import, you could select the text of one or more JTextFields with such a line $(“.HTextField”).as (HTextField).selectAll(); (Tip: If you’re using Eclipse, static imports can be added on demand, type $(“.HTextField”).as (HTextField.HTextField), select the HTextField variable, and then ‘add import‘).

Quite readable, short and clean, IMO. And since the query goes straight to Shizzle, of course all of its syntax is supported, which is pretty powerful.

The last method is a work-in-progress, and you can’t really use it just yet. Like jQuery, hQuery is really small, but the point is to have a tiny core that can be extended with plugins. The with() method takes a class extending the last of the 2 classes contained in hQuery (the DelegatedHQuery class), there’s some useless examples in the code, we’ll see what happens when we cross that bridge but the first plugins that came to mind were related to JXLayer, effects, animations and transitions, form validations, etc.

Shizzle and Composites

Since hQuery binds Shizzle and Composites together, under a nice simple interface, I’ve updated both versions. While Shizzle saw some little modifications, a couple more tests, and so on, the composites have changed quite a bit (even if I surely intend on recoding them from scratch once again, so they’re better tested, right now, it’s pretty well, organic).

The composites I’ve already described allow you to have any number of objects behave like one object (within certain technical restrictions). The version I released was based on the JDK’s dynamic proxy. The biggest advantage to this is you don’t need another library to use them. The big problem is it only works with interfaces. Since then I’ve switched to cglib to create composites of classes (this is where the technical restrictions come into play, as final classes and methods can’t be easily used in a proxy). It also comes with the perk that the proxies are faster than the vanilla JDK ones. The downsides are, there’s another dependency, and since we’re generating proxy classes, the PermGen could become polluted (even if in practice this has never happened to me, I have not looked at it that closely since today’s project only creates a low number of proxy classes. If that problem ever surfaces, Google’s Guice already contains solutions which can be looked at, and Guice also uses cglib, so it’d be fairly easy to integrate).

Composites are at the lowest level, you don’t create them or need to think about them, except in those specific cases where they technically can’t work: creating a proxy on final classes, calling final methods on a proxy, trying to use getters returning classes that can’t be put into another composite.

Another modification since last time is of special interest in the HTk method chaining. If you’re calling a method that returns something, a composite of the results of those method calls will be created (with the regular composite limitations) so you can call another method if you want to. (You can also extract the individual values if you want, look at the code)

What can be improved

Even if this preview is quite usable, and we do use it, there’s still some rough parts that can be improved. As I mentioned before, there’s little support right now to create custom components based on the HTk APIs. It should be much easier. Maybe instead of using client properties for each of the custom components properties, the full HComponent wrapper could be stored inside a client property. Maybe other self referring generics tricks could be used to help in creating classes extending from existing HComponents, like HPanel. Right now, you’d need to override a lot of methods just to return your own type to allow transparent method chaining mixing all the classes.

The different borders methods in HComponent are a little too numerous, I’d like to reduce them to one or two, and move the rest to an inner builder object – maybe component.border().top (5).bottom (2) – or an outer builder – component.border (Borders.empty().top(5).bottom(2)). You guys probably have better ideas.

What needs to be done or would be nice to have

The whole event mechanism is only at its beginnings. All the Swing event classes need EventSources. A consistent good way to use and create custom events should be decided.

Another nice one related to eventing could be live events. Something like $ (“JButton”).live (MouseEvents.click, listener). Basically, you insert your event listener as an AWTEventListener directly to EDT, and this listener would use Shizzle to see if the source matches the query you specified. So whether the UI component tree changed, if new JButtons were added or removed from the UI, you wouldn’t need to add another click listener. It seems cute and nice, but I’m not really sure it’d be as useful as in the HTML DOM (where events bubble and come back down already, which is not the case in Swing when there’s no registered listener for a specific event type)

And of course last but not least, the plugin system would also be interesting to have.

Move to github ?

Downloads

As always, the license in this preview is BSD.

Here’s the zipped Eclipse projects, containing both the source and the binary jars.

Conclusion

I hope this preview can start an discussion, be it about Swing pain points we could solve or even maybe JavaFX 2 (as it seems from the little slides I’ve seen they intend to keep the JavaBeans get/set rule; the only thing that seem different was about properties and events, where it seemed they would use a very low number of methods; I hope the awesome Alex Ruiz is being listened to up there, because he’s the man when it comes to small, powerful, concise and expressive APIs, just look at FEST and dare say I’m wrong).

I hope you’ll find this useful. To me, the combination of small and discoverable set of common methods HTK offers, coupled with composites and CSS selectors make a powerful set of tools.

I’d love to get feedback on this, you can find me at twitter.com/lqd.

Introduction

Welcome to part 2 of our experiments on hardware accelerated effects in Java. Today’s weather forecast: 70% chance of… pixel shaders.

Remember last time, when I said I wouldn’t probably do custom effects and would wait for Sun to provide it to us ? Well…

I lied.

Actually, I changed my mind, but let’s not argue semantics here :]

We’ll see two things in this article. First, we’ll generalize what we did last time and get something usable from plain Java2D, and use that in our Swing support classes. And secondly, I’ll show you how to make your own custom effects/pixel shaders in GLSL & HLSL.

Before we start, a disclaimer: like a lot of the things I do around here, this is going to use some internal APIs, so 1) they might change in the future, 2) I might use them badly (as I think I did in the previous article, it should be cleaned up now hehehe) – 3) there are some things I don’t understand in the APIs, after all there’s so much I can do with undocumented bytecode and without the source really, 4) all this is possible thanks to an ugly hack I won’t bother explaining, because it’s not really interesting.

That being said, I did my best to hide the ugliness behind easy to use API. I can’t be sure this code will work in future releases of Decora, or if it would be doable to adapt it then, though. If that happens, let’s hope Sun provides an officially supported way for custom effects and pixel shaders (which could be as simple as giving access to decora’s compiler and JSL, as I said last time).

Java2D & Swing

There are now 3 ways to use decora effects: 2 for Java2D, and one for Swing with help from JXLayer (which, of course, uses the Java2D ones).

The first one is DecoraEffectRenderer and allows you to draw an Image with an effect applied, on a specific Graphics. This class delegates rendering to some decora utility methods I found not long ago (and might not have been present in decora when I initially wrote this experiment, last december), and renders manually when those can’t be used (“context hijacking”, which I use in the JXLayer support – ie when applying effects on non Java2D-managed images). It contains 3 or 4 drawImage methods (that mirror the ones in Graphics2D). The second one is DecoraEffectImageOp, which as the name suggests is a BufferedImageOp implementation. Those two classes should help you use decora almost like you use Graphics and software effects today in Java2D, so pretty familiar territory for you if you’re reading these lines. The last class is the JXLayer DecoraLayerEffect, from last time, but now just uses a DecoraEffectRenderer internally.

Custom effects / pixel shaders

And now on to the good stuff. Even if the basic concept was easy to code after I got the idea on how to do it, the longest part was coming up with a coherent API I wouldn’t mind using, making sure it worked with GLSL, HLSL, on windows and linux (both using java 6), and so on (unfortunately i couldn’t test on a Mac, I have hopes it will work there, but I have no idea – I think the closest i can get to the Mac environment is java 5 under linux, but decora/JavaFX only run on java 6 there, since they rely on the RSL).

I also want to note the work I did here needs signing in webstart, a requirement I have high hopes of removing when Decora/JavaFX 1.2.1 is released, because of a bug that was recently fixed there. Unsigned webstart is not really a hard requirement for me, but it might be for you. In any case, the method I use for hijacking shaders into decora works unsigned; some files will need to be moved around, but I think that’s basically it. We’ll see how it goes when I can test it, and see if i’m right.

As I said last time, decora has a number of backends because it runs on quite a diverse set of platforms/hardware/software/gpu drivers/etc. What I did here is basically a small subset of what the decora compiler does: only for the gpu backends, and requires manual coding. With JSL you’d code a single file describing your effect, and have the compiler generate all the support classes; here we’ll need to do “everything” manually, and I’ve succeeded in making this task pretty small: one java file, and two pixel shaders, for OpenGL & Direct3D – and of course one of those might be optional, depending on your target platform. For an effect called X, you’ll have in the same package X.java, X.glsl and X.obj (compiled from an HLSL file, using the fxc tool that comes with DirectX SDK)

Let’s see how to make a custom effect. We’ll stay at the global overview level, assuming you can read my code, samples and demos for down and dirty specifics if need be.

As JFXLayer has shown, we’ll need a class extending decora’s Effect. Here to create your custom effect, you’ll extend a helper class I did, called, you guessed it, CustomEffect. And it’ll basically look like this:

Custom effect structure

public class BlingBling extends CustomEffect<BlingBling>
{

// TODO: insert bling here

// ----- Bling controlling -----

@Override
protected Class<? extends ShaderController<BlingBling>> getShaderControllerClass ()
{
        return BlingBlingController.class;
}

public static final class BlingBlingController
   extends Abstract/*Stuff we'll talk about later*/ShaderController<BlingBling>
{
    public final void updateShader (BlingBling effect, Shader shader)
    {
        shader.setConstant ("bling", effect.getBling());
    }
}
}

Hello less-readable-but-safer-because of/thanks to-generics code. Most of this will be generated by your IDE anyway.

The BlingBling java class is the effect implementation. You’ll create instances of this, pass them to DecoraEffectRenderer or DecoraLayerEffect, set parameters, variables and so on. The inner class is the shader controlling part, ie setting variables on the shader, whose values are coming from the BlingBling effect itself. Pretty clean and simple for something that’s quite complicated if you think about it really.

Shader controlling and the shader itself

The shader controlling is grouped by the number of samplers the shader uses: 0, 1 or 2 – because that’s the way decora does it. The shader controller will most of the time extend Abstract[NumberOfSamplers]ShaderController: AbstractZeroSamplerShaderController, AbstractOneSamplerShaderController, AbstractTwoSamplerShaderController. The whole hierarchy is more complex, there are other abstract classes and interfaces you can extend/implement, those are the ones that provide the most help, but you can read the code and figure this out yourself, it’s pretty simple.

Let’s start with what’s common between all of them, the shader structure itself. I won’t obviously explain anything about coding pixel shaders here, and only focus on what’s needed to use and create your own.

I’ll say that there’s more flexibility when using GLSL code – because they can be parsed, composed, and messed with at runtime, which is not easily doable with HLSL – and I’ve called this “prettify-ing” the shader. A “pretty” decora GLSL shader will look like this, a regular GLSL shader:

// bling related variables and samplers

void main()
{
 // bling computing using the mentioned variables and copious texture2D calls

 gl_FragColor = bling;
}

While a real decora GLSL shader actually looks like this:

uniform float jsl_pixCoordYOffset;
vec2 pixcoord = vec2 (gl_FragCoord.x, jsl_pixCoordYOffset-gl_FragCoord.y);
uniform vec2 jsl_posValueYFlip;

vec4 jsl_sample (sampler2D img, vec2 pos)
{
    pos.y = (jsl_posValueYFlip.x - pos.y) * jsl_posValueYFlip.y;
    return texture2D (img, pos);
}

// bling related variables and samplers

void main()
{
 // bling computing using the mentioned variables and copious jsl_sample calls

 gl_FragColor = bling;
}

As you can see there’s a little more boilerplate, and the helper classes basically turn the pretty GLSL shaders into the regular ones. Prettifiying GLSL shaders is optional (but is set to true by default), and also comes with a perk, you don’t have to tell the shader controller the names of the samplers (unless you want to or need to control the IDs), something you have to do with regular decora GLSL & HLSL shaders.

The HLSL shader will look like this skeleton:

// bling related variables and samplers

void main (/* TEXCOORD0 UVs for each sampler */, in float2 pixcoord : VPOS,
   inout float4 color : COLOR0)
{
 // bling computing using the mentioned variables

 color = bling;
}

This is how you’d compile that last HLSL shader to an obj (you could use another file extension if you wanted to, like the more common .ps – but decora uses .obj and that would be mandatory for sandboxed access, so I’ve let it be the default): fxc /nologo /T ps_3_0 BlingBling.hlsl /Fo BlingBling.obj

Now you know why decora (and thus JavaFX) requires a graphics card that supports at least the Shader Model 3.

*-sampler shaders

Let’s look at the different types of supported shaders & shader controllers, what they are useful for, and talk about the included samples, which will show you the exact pixel shader structure you need to follow here.

One-sampler shaders are probably the most common ones, as they represent the kind of effects used in image processing. I provided a sample effect, a basic clone of the SepiaTone decora effect, imaginatively called SepiaToneClone. [Abstract]OneSamplerShaderController offers a way to get the sampler’s name, and whether it is using bilinear or nearest neighbor filtering. To give you an idea, in Decora, the blur, brightpass, and color adjustment effects are all effects using one sampler effects. These shaders would be useful if you wanted to implement some image processing effects for instance, such as the ones you find in Jerry Huxtable’s jhlabs filter library.

Two-sampler shaders, most of the time either mix the two source images, or use one as parameters to apply an effect to the other input image. [Abstract]TwoSamplerShaderController also provides a way to get the samplers’ name, and the filtering to use. Examples of these in Decora are the various Photoshop Blend modes and displacement map effects. I provided a sample effect for that too, which is a clone of the multiply blend mode. These shaders would be useful if you wanted to implement transitions effects for instance, such as the ones you can find in Jeremy Wood’s transitions/transitions2d library.

And lastly, zero-sampler shaders. These, I think, would be the kind of shaders you’d use for procedural textures, like the ubiquitous checkerboard or brick patterns, or the Mandelbrot/Julia fractals for instance, up to the more modern procedural shaders you see in some farbrausch productions (like .kkrieger/.theprodukkt), the game Spore, or the Substance Air tech from my compatriots Allegorithmic. In practice, however, and the reason why I just said “I think” is because you’d probably need UVs for anything worth looking at, and I’m not sure if that’s provided in decora here, since there are no zero-sampler effects in the whole library. That’s why I didn’t provide any sample here, and I started doing my own procedural shaders using one-samplers. So your mileage may vary here.

The java code for the effects, shader controllers & glsl and hlsl shaders (provided in the zip at the end of this article) will probably need to be looked at in more detail to really be able to make a custom effect, but the basic principles have all been described here, and you should be good to go.

Animayshion, man

While the pixel shader support is blazing fast for static images/UIs, be sure to test your target platforms thoroughly if you’re intending to animate those effects, there’s a lot of image data moving from the cpu to the gpu and back here, complicated effects become slow on big images or live UIs because of that. With great power comes great responsiblity, as the great philosopher once said – or maybe it was spiderman, i’m not sure.

I think/hope that the Prism renderer, coming in the next JavaFX release, will solve that. The public info on this is that it will add image caches for any node in the scenegraph: those being on the gpu, animations and pixel shaders should be faster. I’m not sure however that it will allow custom shaders, I doubt it to be honest, and this is part of the reason I changed my mind and coded it myself, however bad and inefficient my code would be – taking into consideration the conditions under which I’m doing this. Plus, Flash (in a sense) and Silverlight already offer this, and I find this lacking in the Java world – Java2D/Swing, not JOGL obviously.

JavaFX support

As for JavaFX support, I’ll start with another disclaimer: I don’t do JavaFX script at all, so take this code with a pinch of salt; if anyone wants to improve it, or correct it because it’s most likely bad, let me know (not to mention I mostly succeeded in crashing the openjfx compiler with those 10 lines). It works like this, you do what’s needed for general java support, and also create a JavaFX class extending my javafx.scene.effect.custom.CustomEffect, to specify and control the custom java effect. This will allow you to use your JavaFX effect like the regular ones.

An example JavaFX effect using the SepiaToneClone sample effect:

package javafx.scene.effect.custom.sepia;

import javafx.scene.effect.custom.CustomEffect;

public class SepiaToneClone extends CustomEffect
{
    public var level : Float = 1.0 on replace
    {
        delegate().setLevel (level);
    }

    protected override function createDecoraEffect() : com.sun.scenario.effect.Effect
    {
        return new org.hybird.decora.effect.sepia.SepiaToneClone();
    }

    function delegate() : org.hybird.decora.effect.sepia.SepiaToneClone
    {
        return decoraEffect as org.hybird.decora.effect.sepia.SepiaToneClone;
    }
}

Crappy demos

I can’t write an article without demos, you know that by now. Unfortunately, they won’t be particularly impressive today, since the point of the article is to show how *you* could do pixel shaders. The sample effects I coded, being mostly clones of existing decora effects, won’t be really new obviously. In any case, here they are, the first one uses the decora renderer to apply a clone sepia color adjustment on a screenshot of the displacement map test demo I talked about on twitter.

The second uses the JFXLayer decora effect and is once again a simplified version of JXLayer’s LockableDemo. Here, a gaussian blur & the clone sepia color adjustment are applied on Swing UI when locking the panel, using the blend multiply clone.

And the last one, is a JavaFX text label over a blue rectangle to which the sepia effect is applied. Clicking will lower threshold. Seriously impressive stuff, i expect George Lucas to call me for help on the next Star Wars, based purely on the aesthetic of this last one. True story.

(Famous) Last words + Download

Of course I wanted to provide more polished demos, with a GLSL editor to play with, for instance, but this article is already getting too long, and I don’t have any more time to make it shorter, or longer – I want to finish it quickly to keep my incredible one-post-a-month average.

Some time in the future we’ll probably see how to make a custom effect that’s actually useful, most likely from one of the “These would be useful if” ones I mentioned here, a transition for example. *Those* would be/will be worth demoing.

You can find the whole project here, with all the support classes, samples, shaders, etc (the whole shebang is under BSD as usual).

Till next time, take care. By the way, you should follow me on twitter here.

PS: this is why I only post once a month, I think I actually aged while writing this 2500-word long article :]

Update (August, 31st): I released version 0.2 of this library, fixing a couple of problems, and adding the possibility of making your own custom pixel shaders. You can find the second part of this series on pixel shaders here.

The Olive Branch

A lot has been said about Swing and JavaFX recently, in a seemingly unending flame war between fans of each side. Both have merits and flaws, and as you’ve seen over time, i’m advocating a 3rd way, which i decided to pompously call The Olive Branch for no reason whatsoever, which is using the best of both worlds.

The plan is to use, directly from java, scenario and decora -the graphics libraries that power the javafx runtime, and which i love probably more than my girlfriend, but don’t tell her (: – (btw if you don’t like crappy jokes, you better stop reading now, and grab a taco or something) . Of course, things have changed since their release, what was once an open source public API (albeit destined to change heavily) is now an internal proprietary one, which means danger, will robinson. So, the plan is to have the community build things that are so cool, and innovative, that Sun will allow us to do just that and make scenario/decora public and usable from java, as they said they would, like with a jnlp extension or something. I think when things have stabilized a little, they’ll get to it. But at the moment, i have succeeded in doing that, at a level of 2% (and i’ve mailed Sun telling them everything i’ve done with scenario in the last 18 months – which is quite a bit in retrospect). Let’s see if we can crank that baby up.

Introducing jfxlayer

I’ve started this a long time ago, i wanted to help people use scenario and decora in swing, not only in a pure scenario scene as with Scenile, but from regular, real world swing, because some tasks are better handled by scenario and decora, and vice versa. The best of both worlds. I never got around to releasing it because i haven’t gotten to a satisfying point in this endeavour (+ i’m lazy), i’ve just planned the whole shebang, and coded the first little step, how to use hardware accelerated decora effects in plain old swing. It’s like 30 lines of code, that i kept in sync with every JavaFX release. I’ll get to the other planned features eventually, in the meantime, here’s the result of this piece of work. Hopefully, it’s interesting enough for regular Swing java users, or more “exotic” ones, like them groovy folks (:

It’s almost nothing code and work-wise. However, this opens a *huge* realm of possibilities for swing apps, and i hope you’ll see that too.

The concept is simple: add decora’s effects to the BufferedImageOps effects that jxlayer allows.

Decora. OpenJFX. JXLayer. JFXLayer.

Decora ?

Decora is Chris Campbell’s baby, if you don’t know who that is, you can stop reading now, bye. This library is arguably the smallest, the easiest to use and most powerful one you’ll ever come across if you’re dealing with effects, image processing, and shaders. A decora effect is made with a JSL file (Java Shader Language) which is basically a mix between Java (in order to get/set parameters and control the effect) and GLSL/HLSL ie, a generic high level OpenGL/DirectX shader language. This JSL file is then “compiled” (by a code generation utility) into a specific effect for each supported backend. The backends are mutually exclusive branches of code that target a specific architecture/3D API, etc. Depending on your software and hardware configuration, decora will choose the backend that will perform the best on your computer, whether you’re on windows mac or linux for instance. You have a regular “software” backend in pure java, one using JNI, one targetting OpenGL, another one OpenGL ES2 for mobile devices, another one Direct3D, and one targeting Prism, the next gen JavaFX graphics renderer, which will allow 3D and more perf, which is about as far as everyone knows. Back in the open source decora days, i’ve also written a backend myself, which multithreaded the regular Java backend, so that effects would use all your available cpus/cores.

All that from one tiny JSL file, they really are small, check the old open source version you’ll see. For the user, it’s of course transparent, you create an instance of the effect you want, and use it, decora handles the rest behind the scenes.

Sure, there aren’t that many effects yet (in fact there haven’t been any new ones since it was released, the work is done on the architecture, perf, backends, for the existing effects, which is very logical if you think about it) compared to, say, what Jerry Huxtable offers in his jhlabs filters library. In practice, it’s enough, for instance you have blurs and shadows, a perspective effect (think coverflow), color transforms, etc and a way to mix and chain all of the effects. If we had access to the compiler (or if we reverse engineered and forward ported that to the open source version) we could add new effects, but it’s not the case anymore. I think it’s doable if you really want it, especially for GLSL shaders which’ll be very interesting once Prism is released, if the compiler isn’t available at that time. I haven’t felt the need to create new effects until this week where i needed a better displacement effect than the one provided. So we’ll see how it goes but i’m not planning on doing it anytime soon.

How do i use it in swing ?

Enough about decora. In your Swing app, use JXLayer, and use the JFXLayer effect bridge with stock decora effects. It’s very simple:

BufferedLayerUI ui = ...; // your regular JXLayer LayerUI
Effect decoraEffect = ...; // your regular decora effect
// like GaussianBlur, Bloom, DropShadow, etc
ui.setLayerEffects (new DecoraLayerEffect  (effect));

Compare that with using JOGL and pixel shaders directly as Romain did in the Filthy Rich Clients book (the sample is like 500 lines long, and which are mostly low level code for setting everything up for rendering, here it takes 5 at most, thanks Chris!)

Demos. Finally! I’m as tired reading this awfully long blog post as you are writing it

I know you’re all about demos and eye candy, and today, i have 3 of them for you. Not one, but three, and if you call right now with your credit card number, i’ll add a set of Ginsu2000 knifes for only 3 easy payments of 9.99$, they can cut a shoe, you know ?

Since this post is about hardware accelerated effects, you’ll obviously need a decent graphics card with up to date drivers.

The first one is pretty basic, it’s two buttons. Pushing each one starts an animation (using the same code that powers the eMotionBlur demo of last time) that progressively blurs the component (but remains live). The button on the left is using a jhlabs blur filter, while the one on the right is using decora’s one. At the default size, it’s really close, but maximize the window and you’ll see the difference, and possibly start to envision the possibilities this offers. Who knows maybe it can be used to power screen transitions (even though i’m still not fully satisfied with the perf yet – maybe Alexander Potochkin, or Dmitri Trembovetski can help us out here, pajalusta (or whatever it’s spelled in cyrillic (: – you’ll see it’s decent enough for me to release it) – Mark, you can keep us updated on that (:

The second one is a modified demo from the JXLayer distribution, using the LockableUI, here also you can compare blurs. On my machine it’s almost instantaneous with decora.

And finally the 3rd one is a modified version of Twimber, my tiny twitter provider for Kirill’s great Amber project. I’ve hacked animated blooms, blurs, and color transforms into it in a matter of an hour, where it would have probably taken me days fiddling with jogl. Bear in my mind this demo is *heavy* with animated effects, and absolutely not optimized at all (for instance the repaints are too big, the effects target components that are too big, etc) and might run slowly (it does a little on my machine) or quite possibly crash your vm (which it does on another machine, probably a bug in my graphics card driver that crashes everything), plus there are some deadlocks somewhere between Amber and the latest releases of Trident, that i haven’t tried to remove, maybe Kirill can help us with that, if need be. If that happens, you can only close the webstart window by killing the process, sorry about that, guys. Hopefully, we fixed the problem, it stll won’t be optimized though ^^

Is it safe to use ?

Sure, using internal APIs is going to be a pain in the neck, it already has been for me. But, all in all, Decora’s API did not change much in the last 18 months, and especially not the effects’. And as the saying goes, no pain no gain. Plus they work fine with JavaFX 1.2 and JFXLayer right now, you don’t have to use the next version if you don’t want to have to modify them.

Downloads and source

Update (August, 31st): As mentioned earlier, I released version 0.2 of this library, fixing a couple of problems, and adding the possibility of making your own custom pixel shaders. You can find the second part of this series on pixel shaders here. I didn’t change the code linked here for archive purposes though.

You can find JFXLayer here (zipped source project + demos), and twimber2 here (zipped source project).

Let me know what you think, if the demos run well, or whatever in the comments, or sooner on twitter.