Scripting News in RHQ 4.5.0

RHQ 4.5.0 (which we released today) contains a great deal of scripting enhancements that I think are worth talking about in more detail. In my eyes, the changes make the scripting in RHQ ready for serious use.

CommonJS support

This, I think, is huge. In the previous versions of RHQ, the only way of reusing functionality from another script was to use the exec -f command in the interactive CLI shell (in another words, this was NOT available in the batch mode, which is how majority of people are using the CLI). So if you needed to implement something bigger and needed to split your code in several files (as any sane person would do), you only had 1 option – before executing the “script”, you needed to concatenate all the scripts together.

This sucked big time and we knew it 😉 But we didn’t want to just add functionality to “include files” – that would be too easy 😉 At the same time it wouldn’t solve the problem, really. The problem with just “including” the files into the current “scope” of the script is that this would mean that each and every variable or function in those files would have to be uniquely named because javascript lacks any sort of namespace/package resolution. Fortunately, the CommonJS spec solves this problem.

Here’s how you use a module. Notice that you assign the module to a variable and that’s how you prevent the “pollution” of your scope. The loaded module can have methods and variables with the same name as your script and they won’t influence each other:

var myfuncs = require("modules:/myfuncs");
myfuncs.helloworld();

You may wonder what that "modules:/myfuncs" identifier means. It is an URI that the underlying library uses to locate the script to load. This “sourcing” mechanism is pluggable and I will talk about it more in the following chapter. To see some examples of the modules, you can look at the samples/modules directory of your CLI deployment and you can also read some documentation about this on our wiki.

Locating the scripts

With the ability to load the scripts there comes the problem of locating them. For the standalone CLI, the obvious location for them is the filesystem, but what about alert notification scripts on the RHQ server? These scripts are stored in RHQ repositories which don’t have a filesystem location. The solution is not to tie the scripts to the filesystem but have a generic way of locating them using URIs and a pluggable way of resolving those URIs and loading the scripts from various locations. This means that you can for example load the script from an RHQ repository in your standalone CLI installation, or to define 1 central location for your local CLI scripts and use the “modules” URIs to refer to them. Or you can easily implement your own “source provider” and for example load the scripts from your internal git repo or ftp or whatnot. RHQ comes with a small set of predefined source providers, documented here.

With this ability at hand, you can make an RHQ repository a central place for your scripts that you will then be able to use universally – both in the standalone CLI installations and also in the alert notification scripts.

Python

In previous versions, our scripting was tied to Javascript. Thanks to quite a bit of refactoring, the RHQ scripting integration is now language independent and language support is pluggable (see my previous post where I detail how this was done in case of Python).

What this all means is that you can now write your CLI scripts in Python and still use the same API as you were able to use before from Javascript only. I.e. you will find the ResourceManager, AlertManager and all the other predefined variables that define the RHQ API available in Python, too. The only thing that this initial implementation doesn’t support is code-completion in the interactive CLI shell.

Last but not least, the ability load the scripts from various locations is available in Python, too, using an automatically installed path_hook. You can read about how to use it on our wiki. This also means that you can now write your alert notification scripts in Python, too.

When running an alert notification script (i.e. an alert notification of the type “CLI Script”), the language of the script is determined from the script file extension – “.py” for python and “.js” for javascript. When you start the CLI shell, you pass your language of choice using the --language commandline parameter – “javascript” or “python” are the obvious possible values for it.

Conclusion

In my opinion, these changes are great and will allow our users to really start building useful tools using our scripting support. If you feel like you’ve come up with a script module you would like to share with the RHQ community, why don’t you just send a pull request to our github repo with sample scripts?

The Dark Powers of PowerMock

Recently, we’ve started using Mockito and PowerMock in our testing. I won’t explain mocking and why or why not you should use it, but I want to share my experience with using PowerMock.

PowerMock comes with a very strong promise: “PowerMock uses a custom classloader and bytecode manipulation to enable mocking of static methods, constructors, final classes and methods, private methods, removal of static initializers and more.”

That is seriously cool, right? I thought so, too, but I stumbled upon several problems the very first time I tried to use it. Frankly, those problems, as always, stemmed from my lack of experience with the tool, but hey – everyone’s a novice at first. Let me share my experience with you.

The Problem

 
public class ClassUnderTest { 
    public InputStream method(boolean param, URI uri) throws Exception { 
        String scheme = param ? "https" : "http"; 
        URI replacedUri = new URI(scheme, uri.getAuthority(), uri.getPath(), uri.getQuery(), uri.getFragment()); 
        return replacedUri.toURL().openStream(); 
    } 
}

The above fabricated example expresses the essence of the testing challenge I faced (the real class was this.) The method I wanted to test obtains an URI and transforms it based on some parameters. Then it tries to open a stream on the URI so that the caller can download the contents.

Because the URI that the method tries to download from is by design either http or https URL, it is kind of hard to test without actually standing up a HTTP server to serve the file during the test. This is of course not impossible and possibly would not be that hard, but I thought PowerMock can come here to the rescue. I should be able to mock those calls out in my tests.

Attempt #1 – mocking system classes

 
@Test
@PrepareForTest(ClassUnderTest.class)
public class MyTest { 
    @ObjectFactory 
    public IObjectFactory getObjectFactory() { 
        return new PowerMockObjectFactory(); 
    } 

    public void testMethod() throws Exception { 
        URI uriMock = PowerMockito.mock(URI.class); 
        URL urlMock = PowerMockito.mock(URL.class); 
        
        PowerMockito.whenNew(URI.class).withArguments("http", "localhost", null, null, null).thenReturn(uriMock); 
        Mockito.when(uriMock.toURL()).thenReturn(urlMock); 
        Mockito.when(urlMock.openStream()).thenReturn(new FileInputStream(new File(".", "existing.file"))); 
        
        ClassUnderTest testObject = new ClassUnderTest(); 
        testObject.method(false, new URI("blah://localhost")); 
    } 
} 

This should be fairly easy to understand for everyone that used some mocking framework. I’m creating two mocks: one for URI and one for URL classes. Then I’m using PowerMock to capture the construction of a new URI (see the code of the ClassUnderTest) and returning my uriMock. The uriMock is set up to return the urlMock when its toURL() method is called. When the openStream() method is called on my urlMock, I’m returning an input stream of a local file.

Nice and easy, right? Except it doesn’t work. I get the following stacktrace as soon as I try to mock the URI class:

 org.mockito.exceptions.base.MockitoException: Mockito cannot mock this class: class replica.java.net.URI$$PowerMock0 Mockito can only mock visible & non-final classes. 

After a bit of googling, the cause is apparent – PowerMock cannot mock the system classes (unless PowerMock java agent is used). Ok, let’s try another approach, this time trying to avoid using mocks.

Attempt #2 – PowerMockito.whenNew(URL.class)

The idea behind this attempt is that PowerMockito can capture and override constructor calls. Because URI.toURL() constructs a new URL instance with a single string argument, so we theoretically should be able to intercept that?

 
public void testMethod() throws Exception { 
    URL realUrl = new File(".", "existing.file").toURI().toURL(); 

    PowerMockito.whenNew(URL.class).withArguments("http://localhost").thenReturn(realUrl); 
    
    ClassUnderTest testObject = new ClassUnderTest(); 
    testObject.method(false, new URI("blah://localhost")); 
} 

As you might have guessed, this doesn’t work either. And frankly if it did, I’d have some serious questions about how it could. The constructor of URL is only called inside the toURL() of the URI which is a system class that PowerMock can’t touch. So, the third attempt.

Attempt #3 – PowerMockito.whenNew(URI.class)

What is the difference between this one and the previous attempt? Well, it took me a while to decipher the javadoc for the @PrepareForTest annotation, but it boils down to this. If you need to use the PowerMockito.whenNew method, you need to tell PowerMock to do bytecode manipulation on the class that (in some method) directly calls given constructor. This is kinda understandable when you know what PowerMock is doing – it will actually change the byte code of the “prepared” class so that any constructor calls (and other things) are checking for the rules defined using whenNew and other methods. You realize this for real when you try to debug the class under test (that has been prepared by power mock) – you can no longer be sure that what you see in the code is actually what is happening, because the bytecode of the class no longer exactly corresponds to what you see in the source code.

So to sum it up, here’s the code that works:

 
@Test 
@PrepareForTest(ClassUnderTest.class) 
public class MyTest { 
    @ObjectFactory 
    public IObjectFactory getObjectFactory() { 
        return new PowerMockObjectFactory(); 
    } 

    public void testMethod() throws Exception { 
        URI realUri = new File(".", "existing.file").toURI(); 
        PowerMockito.whenNew(URI.class).withArguments("http", "localhost", null, null, null).thenReturn(realUri); 
        
        ClassUnderTest testObject = new ClassUnderTest();         
        testObject.method(false, new URI("blah://localhost")); 
    } 
} 

The constructor of the URI is intercepted and we return a “realUri”, i.e. a different instance of otherwise “normal” URI class. This works, because exactly that constructor with those arguments is called in the class under test that has been manipulated by PowerMock (as instructed by the @PrepareForTest annotation). From that point on, we don’t need any special behavior on either the URI or URL classes and so the code can stay untouched.

Conclusion

The conclusion is basically the famous 4 letters – RTFM 🙂 I just wanted to detail my journey through the dark corners of the PowerMock forest just in case some of you were as confused as I was when I first entered it.

Posted in Java, RHQ. 2 Comments »
%d bloggers like this: