Photo from Chile

Introducing CFSelenium - A Native ColdFusion Client Library for Selenium-RC

We were very lucky to have Adam Goucher, a member of the core Selenium team and the maintainer of Selenium-IDE, as a speaker at the Toronto ColdFusion User Group this month. Adam delivered an awesome presentation about automated testing in general and Selenium in particular.

During the session I asked a bunch of questions about the architecture of the Selenium components, in particular Selenium-RC and WebDriver, which is now part of Selenium 2.0. Through our conversation I came to understand that it would be fairly simple to create a native ColdFusion client library for Selenium-RC, so I decided to give it a try. Thankfully it was as simple as I expected and I decided to complete the library so that the full API of Selenium-RC is supported.

Thus CFSelenium was born and was given a home at GitHub. If you have no idea what I'm talking about, fear not, I will provide some more background information on Selenium and Selenium-RC in the rest of this post.

What is Selenium?

At its most basic Selenium is a tool that lets you automate a web browser, which is mostly used for automated testing. With it you can instruct a browser to do pretty much anything that a user of your web application might do, and you can ask the browser questions about the resulting Document Object Model (DOM), the answers to which can be fed into asserts in tests. Selenium works by injecting JavaScript into the browser which is then used to control the browser. Selenium is free and open source.

What is Selenium-IDE?

Selenium-IDE is a Firefox plugin which can be used to record and playback scripts which automate the browser. It is an excellent entry point into using Selenium as it's extremely easy to set up and use, and you can create complete tests in it that include verifications and asserts. As you can walk through your web app recording actions as you proceed it is a nice way of getting familiar with Selenium commands, and I've used it in very simple tests and one-off tests quite a bit, but when you start getting into serious functional tests you generally want to use an actual programming language to define and drive your tests, not to mention allowing tests to be run on browsers other than Firefox. That's where Selenium-RC comes in.

What is Selenium-RC?

Selenium-RC, which stands for Selenium Remote Control, consists of two parts:
  1. The Selenium Server, a Java application that launches and kills browsers, and controls the browsers by interpreting Selenese commands that have been passed to it.
  2. Client libraries, which provide the interface between each programming language and the Selenium RC Server.

There are some major advantages to using Selenium-RC for your functional tests rather than Selenium-IDE, including:

  • Your tests can now be executed against practically any browser that supports Javascript. Specifically Selenium-RC supports Firefox, Internet Explorer, Safari, Opera and Google Chrome, but you can control other browsers as well using the *custom browser command.
  • You can write your tests using a real programming language which means you can include loops and conditional logic, as well as programmatically generating test data and/or pulling data from a file or database.

Prior to CFSelenium, client libraries were available for Java, Python, Ruby, C#, Perl, PHP, and perhaps others that I'm missing. Although it's possible to control the RC server via ColdFusion code using the Java client library, I thought it would be interesting to see if I could create a library that used only ColdFusion, as this seemed to offer a couple of advantages:

  1. It's easier to get started with as you don't need to find and load the Java driver.
  2. As the cfc implements the complete API, you can get code assist when using the component from within ColdFusion Builder.

Using the Selenium-RC server and the new ColdFusion client library you can now write tests using your favourite ColdFusion testing framework (e.g., MXUnit).

Requirements

Because of the way the cfc was written, you must have ColdFusion 9.0 or above to use it. There is no functionality required that is not present in earlier versions of ColdFusion, but I chose to write the component as a pure script component, which is why the requirement of CF9 exists. It would not be terribly difficult to change the format of the component to work under earlier versions of CF if someone was so inclined to do so.

In order to actually run tests you need to have a copy of the Selenium RC Server jar, but a copy is included with the distribution, in a folder called Selenium-RC.

How to Use It

Step 1 - Start the RC Server

Simply start the server as you would any other Java program. I'm on OS X, so I just open up a terminal window, navigate to the folder that contains the jar, and execute the following command:

view plain print about
1java -jar selenium-server-standalone-2.0b2.jar

Step 2 - Create an Instance of the ColdFusion Client Library

The remaining steps are all code that is likely to exist inside a test method, although steps 1 and 2 might be in setup() and step 5 might be in teardown(). You simply instantiate an instance of the selenium.cfc component that comes with the distribution. The only required argument to the constructor it the url from which you wish to start your test. Here's an example:

view plain print about
1selenium = new selenium("http://github.com/");

You can optionally pass in arguments for the host and port on which the RC Server is running (which default to localhost and 4444, respectively) as well as the browser that you wish to launch (which defaults to Firefox). For example:

view plain print about
1selenium = new selenium("http://github.com/","localhost",4444,"*googlechrome");

Step 3 - Launch the Browser

To launch the browser you call the start() method on the selenium object:

view plain print about
1selenium.start();

Step 4 - Control the Browser and Do Asserts

You now work with the selenium object to control the browser and interrogate the resulting DOM to feed data to your asserts. For example:

view plain print about
1selenium.open("/bobsilverberg/CFSelenium");
2assertEquals("bobsilverberg/CFSelenium - GitHub", selenium.getTitle());
3selenium.click("link=readme.md");
4selenium.waitForPageToLoad("30000");
5assertEquals("readme.md at master from bobsilverberg's CFSelenium - GitHub", selenium.getTitle());
6selenium.click("raw-url");
7selenium.waitForPageToLoad("30000");
8assertEquals("", selenium.getTitle());
9assertTrue(selenium.isTextPresent("[CFSelenium]"));

The above test would:

  1. Navigate to the page http://github.com/bobsilverberg/CFSelenium.
  2. Verify that we're on the correct page by asserting that the page title is bobsilverberg/CFSelenium - GitHub.
  3. Click on the anchor tag with the link text of readme.md, and wait for the resulting page to load.
  4. Verify that we're on the correct page by asserting that the page title is readme.md at master from bobsilverberg's CFSelenium - GitHub.
  5. Click on the anchor tag with the id of raw-url, and wait for the resulting page to load.
  6. Verify that we're on the correct page by asserting that the page title is an empty string.
  7. Verify that we're on the correct page by asserting that the text [CFSelenium] is found on the page.

Step 5 - Kill the Browser

To kill the browser you call the stop() method on the selenium object:

view plain print about
1selenium.stop();

You should always kill the browser at the end of a test, pass or fail, so this is a good candidate for teardown().

What's Next?

Testing

I admit to having done only rudimentary testing on the client library thus far, and I've already identified and fixed one bug. I have had a few folks volunteer to help me test the library, for which I am grateful, but I'd love to have more help. If you're interested in giving it a try I'm happy to help you get set up and attempt to answer any questions you may have. Either send me an email or add a comment to this post to let me know if you'd like to help.

A Formatter

I'd like to add a formatter to Selenium-IDE which would allow you to export a script built in the IDE into an MXUnit test that uses the ColdFusion client library. Adam has been giving me some advice on this and I hope to have something in the next week or two. In the meantime, if you export from Selenium-IDE into Java format you can pretty much take the bulk of the generated code and drop it into a test method in an MXUnit test case as is.

Debugging

I'm thinking of adding a debug mode which would automatically capture information to make debugging your tests easier. When turned on this would slow the tests down tremedously, but it might be a nice to have feature for those times that your tests aren't working and you're not sure why.

In Conclusion

I'd like to thank Adam again for his help and if anyone has any questions about, or suggestions for, the client library please let me know.

TweetBacks
Comments
Bummer that it is CF9 only. Many of us are stuck with CF8 or less so we are out of luck. Do you find that writing in cfscript that much more productive? I have been wondering about that, but only have CF9 on my development machine so there isn't much reason for me to move that direction for a while.

Sounds like a great idea and I look forward to more posts about it.
# Posted By RogerTheGeek | 2/22/11 10:02 AM
That looks very promising! I want to jump on the bandwagon ASAP because right now I am "using" MXunit and selenium in parallel and it isn't that much fun. Keep up the good work.
# Posted By Jean Ducrot | 2/22/11 11:53 AM
@Roger: Yes, it's simply a personal preference. It really wouldn't be that difficult for someone else to take my component and create a tag-based version. Feel free to give it a shot yourself. ;-)

@Jean: Please let me know if you do give it a try and whether it works for you.
# Posted By Bob Silverberg | 2/22/11 11:56 AM
This looks very interesting. We've just started going down the path of TestNG framework and Selenium RC. Tests are recorded in Java format in Selenium IDE, then made dynamic in TestNG Java classes where dataproviders and custom logic add flexibility and robustness to the tests. Would be great to use MXUnit and CFML to build out tests. I'll definitely be looking at CFSelenium. Thank you.
# Posted By Steven Erat | 2/22/11 1:20 PM
Looks awesome. I'm very interested in this project. Keep it up! I'll try it out as well soon.
# Posted By Sami Hoda | 2/22/11 2:44 PM
Ditto what Sami said.
# Posted By jalpino | 2/22/11 3:25 PM
Excited :) Well Done!
# Posted By Brett | 2/22/11 3:31 PM
Thanks everyone, I'm eager to hear your feedback.

@Steven: I'm mostly done a formatter for the CFSelenium client library so you could do exactly the same thing: record you tests using Selenium-IDE and then export in CFSelenium format complete with an MXUnit test case wrapper. I should have the first iteration done this week if you want to give it a try.
# Posted By Bob Silverberg | 2/22/11 3:46 PM
Nice one Bob!

I can't wait to give this a whirl once the formatter is complete. Sounds great!
# Posted By Mark Mandel | 2/22/11 5:47 PM
Re CF8 or CF9 - I'm not sure how well this translates in the CF world, but there is nothing that says you can't build a site in CF8 and script it in CF9. Too often people script in Java because their application stack is in Java when it is a much nicer experience to do it in Python or Ruby
# Posted By Adam Goucher | 2/22/11 5:55 PM
@Mark: There's actually already a Firefox extension in the distribution (which I built today), I just haven't blogged about it yet.

@Adam: What you say is true, but I think there are lots of folks who only have access to CF8, so it's not likely they'd be writing tests on CF9 to test CF8 web apps.
# Posted By Bob Silverberg | 2/22/11 10:14 PM
Hi Bob,
This is pretty awesome.
I have always wanted to get into Selenium ever since I saw Peter Bell at the WebDU conference explain how to use it as a "macro" for speeding up workflows that you use all day long. (like logging into an application).

I have a rudimentary play and conform that it works as advertised.

Now I just have to go and learn all the Selenium commands!

I'd be happy to volunteer any testing you might need, too.

Beau.
# Posted By Gavin Baumanis | 2/23/11 12:01 AM
@Bob Thanks, I'd love to try out the CFML formatter with MXUnit. It would provide much more flexibility to my tests since my Java skills aren't at the same level as my ColdFusion skills. Anything I can do to help out please let me know. Will be OOO most of next week while at SOTR.
# Posted By Steven Erat | 2/23/11 12:32 AM
@Steven: The Firefox plugin is now available: http://www.silverwareconsulting.com/index.cfm/2011...
# Posted By Bob Silverberg | 2/23/11 10:27 AM
Thanks Bob, I'll give it a spin perhaps later today or tomorrow.
# Posted By Steven Erat | 2/23/11 11:32 AM
I built a practically identical CFC (which I called SeleniumTester to match naming convention in the extended MXUnit framework that I use). A word of warning for when you get ready to mix up MXUnit calls with Selenium RC calls: if you are using rollback with MXUnit, as soon as you create a record within the transaction block, you'll find that your Selenium tests will fail with a timeout because the table is locked for rollback. I tried creating my test records within a transaction block that had isolation="read_uncommitted" -- but for some reason the table was still locked when I tried to access it from a page called by Selenium later in the test.

Still looking for a solution if anyone has any ideas. I use Steve Bryant's DataMgr (http://www.bryantwebconsulting.com/docs/datamgr/) and he is currently looking into storing and accessing data somewhere other than the DB during unit tests to overcome this, but that will obviously take time to implement. I may have to resort to tracking and deleting any test records at the end of the test if I don't find another way around it soon.

Also, if you want to get fancy, you can set up Selenium RC as a Windows service so you don't have to start it manually each time you want to use it. I used the following as a reference and got it going easily: http://salmontech.blogspot.com/2009/09/running-sel...

Caveat: sometimes the service stops working and has to be restarted. It doesn't actually "stop," it just stops accepting connections. Also happy to hear any suggestions on solving that one. Has me stumped.
# Posted By Tim DeMoss | 2/23/11 12:49 PM
I would love to see a presentation on this, even a short vimeo video on setup/use/mxunit
# Posted By Dan Vega | 2/23/11 1:02 PM
Hey Bob, it's working brilliantly. Thanks so much. I recorded one long test with the CFSelenium IDE, pulled that into MXUnit, and it runs great.

In TestNG I had been creating expressions like this:
selenium.waitForCondition("selenium.isElementPresent(\"//input[contains(@id, 'InsuredName')]\")", "10000");
      
But that syntax didn't port directly to ColdFusion, so I ended up writing a custom function instead like this:

// wait for Javascript to create the new text input fields
selenium.waitForCondition(generateCondition(element="input",attr="id",operator="contains",attrVal="ClassCode"), 10000);
      
private string function generateCondition(required string element, required string attr, required string operator, required string attrVal){
      local.target = '"#arguments.attrVal#"';
      local.xpath = "'//#arguments.element#[#arguments.operator#(@#arguments.attr#, #local.target#)]'";
      local.condition = "selenium.isElementPresent(#local.xpath#)";
      return local.condition;
}

Also, I'll be thinking about how to architect the test suite for best code reuse. For example, TestNG has annotations for @BeforeTestSuite and @AfterTestSuite in addition to @BeforeTest & @AfterTest. Presently I'm running selenium.start()/.stop() in the suite-level annotations rather than in every testcase. I'm also thinking about whether or not to just directly port the JExcelAPI strategy for spreadsheet-based DataProviders we're using in TestNG or just go with CFQuery and a test database.

But I'm excited that I can initiate the test via a mxunit ant build file or run it directly from the mxunit plugin. And I'm most excited that now I can utilize the full power of my CF skills to make the tests more flexible.

Very cool. Thanks so much for your work.
# Posted By Steven Erat | 2/24/11 10:00 PM
Thanks for all the feedback Steven. Here are some questions/ideas:

1. Do you think it makes sense to add that generateCondition() method to the tool somehow? Either write it out to the test case in the formatter or include a baseTestCase.cfc with the tool? Also, assuming that we do make it available, would it be practical to generate calls to it via the formatter, or is that code that is not generated in Selenium-IDE and is added after the fact in your TestNG tests?

2. Did you know that MXUnit now has beforeTests() and afterTests(), which I'm guessing are the same as TestNG's @BeforeTestSuite and @AfterTestSuite? Could you use those in the same way? I can see value in only starting and stopping the browser once for the entire test case. Maybe the formatter should be changed to use those instead of setup() and tearDown()?

3. Are you familiar with MXUnit's dataproviders? I wonder if you can use those in a similar way to how you're using the JExcelAPI strategy?

Maybe these are topics we should discuss on the MXUnit mailing list to get input from others as well?
# Posted By Bob Silverberg | 2/24/11 10:14 PM
I'll be researching what mxunit has to offer for beforeTests/afterTests, and mxunit dataproviders today. I wasn't aware it had those features.

Regarding the generateCondition, that was my first pass at solving that problem. The waitforcondition method takes a String as its argument, then evaluates and runs that String as a method. The nesting of quotes fouled me up, so that's why I used the custom function. I'd have to play with it some more to determine if its a good standard practice or if there's a better way.

I'll probably port our TestNG suite over to mxunit since we are in the early stages of suite development and now would be a good time to do that.

I'll check the mxunit blog to see if there's a technical discussion going there about this.
# Posted By Steven Erat | 2/25/11 10:57 AM
I upgraded my Selenium Server from v1.03 to v2.0.0 and now my UI Test Suite runs again. Whew. Firefox 5 officially supported.
# Posted By Steven Erat | 7/11/11 12:07 PM
Can you please update it for FF 5.0.1 ? i would really appreciate it. by mistake i updated my FF5.

THanks
# Posted By Manithan | 7/14/11 5:07 PM
Sorry i meant the Firefox plugin
# Posted By Manithan | 7/14/11 5:08 PM
Sorry for the delay. A new version has just been released which supports Firefox 5. You can get it at http://cfselenium.riaforge.org/.
# Posted By Bob Silverberg | 7/27/11 9:06 PM
Thanks very much for all your work on this project. It's so extremely helpful.

I just upgraded to the most recent version (a little slow since I'm coming from 1.1!) and I'm wondering--why is it that waitForElementPresent doesn't use the selenium waitForElementPresent function? Something like:

doCommand("waitForElementPresent",[arguments.locator, arguments.timeout]);

Does that cause unexpected results in some places?
# Posted By Nicole Stevens | 9/15/11 2:40 PM
@Nicole, I don't think there is a waitForElementPresent command available to Selenium RC. Have you been able to get that to work?
# Posted By Bob Silverberg | 9/15/11 4:43 PM
Yes, and I wish I could give you more documentation. This stuff is all new to me so I've been to a lot of sources and it all runs together. Today while I'm trying to go back and see where I saw it, all the sources I find are not for the RC, and they're kind of vague:

http://seleniumhq.org/docs/02_selenium_ide.html#th...
http://release.seleniumhq.org/selenium-core/1.0/re...

For what it's worth, I did try it with the newer version of the RC today and it is working for me there, too.
# Posted By Nicole | 9/20/11 7:50 PM
@Nicole, I think those doc links are for Selenium-IDE. If you find that issuing the command actually works with Selenium-RC then perhaps it is something that has been added. You are free to use it if you wish, but I don't think we'll bother to change the working of CFSelenium as long as it's working fine there.
# Posted By Bob Silverberg | 9/26/11 11:33 AM
Sounds good.
# Posted By Nicole | 9/26/11 5:38 PM
OK, so I am totally new to Selenium and MXUnit but I'm trying to get it setup. I do my development work on an external webserver and so I don't have ColdFusion running on my local machine. Does Selenium open the browser on the server's computer or on my own local computer? I would like for it to use my local browsers to the run tests and display the results. I think it might be using the server's browsers because I'm getting the following error which I think might be because the server doesn't have Firefox installed:

The Response of the Selenium RC is invalid: Failed to start new browser session: java.lang.RuntimeException: Firefox 3 could not be found in the path! Please add the directory containing ''firefox.exe'' to your PATH environment variable, or explicitly specify a path to Firefox 3 like this: *firefox3 c:\blah\firefox.exe
# Posted By Brandon Ruffridge | 1/18/12 3:32 PM
After using my brain, I realized that CFSelenium runs the browsers of the server from which it is run. So after I installed CF9 developer edition on my laptop I was able to get everything setup. I wrote about the experience on my <a href="http://www.linecomments.com/2012/01/web-ui-regress...;.
# Posted By Brandon Ruffridge | 1/25/12 11:24 AM
I'll try to post the link again. This time without html.

http://www.linecomments.com/2012/01/web-ui-regress...
# Posted By Brandon Ruffridge | 1/25/12 11:26 AM
I am fairly new to Selenium. We are trying to venture into the world of System Testing for our CF application. We started with the Java Webdriver libraries but then I found out about CFSelenium while at Steven Erat's excellent presentation at CFObjective 2012. I have CFSelenium up and running but I am having some difficulty understand the implementation.

It appears that CFSelenium uses SeleniumRC via HTTP calls to the API. From what I understand this is the "older" API for Selenium and Webdriver is the "future". Is this correct? If so, is there an advantage to using Webdriver (via Java). Is there any consideration (or advantage) for CFSelenium wrapping the WebDriver API instead of supporting SeleniumRC?
# Posted By Dirk Meilinger | 5/30/12 1:35 PM
@Dirk: CFSelenium does only currently support Selenium-RC and the "old" Selenium 1 API. If you are going to use page objects, as Steven described in his session, then you can abstract that API out, and even if you later update to a Selenium 2/Webdriver style API your tests could remain unchanged.

There are no current plans to create a version of CFSelenium that supports Selenium 2/Webdriver, but contributions are welcome if it's something you'd be interested in doing.
# Posted By Bob Silverberg | 5/31/12 12:33 PM
@Bob - It would be really helpful to be able to leverage the webdriver features, too.
I recently ran into a problem with an ajax form submission. Works fine manually, works fine through selenium IDE, but just hangs running through selenium RC (via CFSelenium).
I inspected the HTML request responses (manually - via net console in FF, through CfSelenium/selenium RC - by logging the captureNetworkTraffic contents).

When running manually, there are between 4 and 13 request/responses on button click and they all look normal. When running through selnium RC, the first request/response includes http status code 411 (length required). Oddly enough, this doesn't trigger the "invalid response from server" error. it just hangs.

Found this link about a very similar issue, but no resolution:
http://code.google.com/p/selenium/issues/detail?id...

Let me know if you have any ideas about how to get around this.
# Posted By Heidi Smith | 7/26/12 12:36 PM
Based on the comment in this link stating that they didn't get the error when running in *iexploreproxy, I tried it.
Sadly, after setting up to use *iexploreproxy instead of *firefox, the application behaves the same way.
# Posted By Heidi Smith | 8/1/12 4:53 PM
oops - didn't include the link in the previous comment:
http://code.google.com/p/selenium/issues/detail?id...
# Posted By Heidi Smith | 8/1/12 4:54 PM