How to scale Selenium

How to scale Selenium?

Introduction

Selenium/Webdriver is a powerful test automation framework for testing web applications. It is the actual standard for web application test automation. 

Selenium/Webdriver is great because of:

• Real world experience: Selenium/Webdriver launches and drives a real full GUI browser to simulate the user behavior. 
• Multiple Browser/OS support: Selenium/Webdriver almost support all popular browsers and OS
• Free: Selenium/Webdriver is free. 
• Technical support: The Selenium/webdriver community is very active. The Selenium is supported by SauceLab. The Chromedriver is supported by Google. 

However, using Selenium/Webdriver is not easy. It is because:

• Flaky test: Selenium/Webdriver may not generate 100% accurate results. Flaky test happens from time to time
• Scalability: Selenium/Webdriver architecture is not scalable to support a big cluster like 10+ nodes and 1000+ tests at the same time. I do not have the data. Please do not quote me. Based on my personal experience, it is already pretty painful to use about 4 nodes and run 30+ test suites at the same time. 

Next, I am going to talk about the challenges to deal with the scalability and flaky test. 

Challenges

Based on my experience, the major challenge comes from two places: 

• Many moving parts
• Not scalable architecture

If you investigate the full lifecycle of a Selenium call, you may find the following path:

Selenium Client -> Selenium Hub -> Selenium Node -> Webdriver (ChromeDriver/SafariDriver/IEdriver) -> Plugin/Extension in Browser -> Browser

A selenium command is a JSON object. It is sent from the client to the selenium hub using HTTP protocol. The hub then relays this request to a Selenium Node. The Selenium Node relays to this request to the Webdriver. The Webdriver interprets this request and converts it to a few calls to the plugin/extension in a opened browser. The plugin/extension in the browser executed the calls to do the real work like extract text from a DOM node or change text of a DOM node. 

Selenium's command/protocol is quite verbose. For each test case, there could be hundreds of Selenium commands sent to the Selenium Grid. You will understand it when you see the console of a Selenium Node. 

Selenium Hub is a sort of bottleneck. There could be only 1 Selenium hub in a Selenium Grid. When it is overwhelmed, the whole Selenium Grid could be impacted. 

Running web application test is also expensive or resource intensive. Each browser session may take quite a lot CPU time and memory when rending web pages and handling client side javascript code. When you run multiple browser sessions at the same time, the Selenium Node may become busy. When the Selenium Node is a little busy (Not need to be 90% or 100%, just when the cpu utilization is about 70%), there could be different random things happening. 

Many moving parts, plus many commands, plus a bottleneck, plus the resource intensive web application test. Do you still expect every test case is run accurately every time? I do not believe so. 

So, what can we do? Obviously, changing the architecture is impossible for users. What we can do is to properly design the Selenium Grid and tune different parts to achieve the best result we can get. 

Good practices

1. Set up Selenium Grids

Using Selenium Grid is the recommended way to scale Selenium. However, as I mentioned, the Selenium Grid itself may become the bottleneck and have a lot of problem. Therefore, I do not believe that it is feasible to manage a Selenium Grid with 10+ Selenium Nodes. If you do have a use case to set up a big Selenium Grid, it may be better to set up a few smaller Grids. 

2. Tune the Selenium parameters

You need to add the Xms and Xmx Java options to increase the Java heapsize. The default 64M heap size is too small. When there are many concurrent tests running, the JVM may do quite a few GC, which will slow down or screw up the tests. A good practice is to use VisualVM to monitor the heap size and GC Activity when tests are running. That way, you will know what the heapsize is appropriate. Do not forget the add Xms and Xmx for the Selenium Hub also. 

You also need to plan the total number of sessions and the number of sessions for each browser type. By doing this, you limit the number of browser sessions a Selenium Node can run at the same time. As a result, you can avoid a Selenium Node to be super busy. To do so, you have to run multiple tests to verify when the system resource utilization reaches the threshold (for example 70%). 

3. Tune the host of the Selenium Node

It is often the case we only focus on the Selenium itself and forget about the host. If the host has 1 CPU and 4G memory, it is not surprise that the tests could be slow and error-prone. Again, web application test could be resource intensive. Make sure you have the proper hardware and keep an eye on the resource utilization. 

Meanwhile, you also need to tune the OS level parameters. I found this because of my recent troubleshooting experience. 

Because Selenium Protocol is "verbose", the ephemeral ports can be used up very quickly. The problem was mentioned by the Selenium/Webdriver team. 

https://code.google.com/p/selenium/wiki/ScalingWebDriver

A good practice is to decrease the Time_Wait timeout time so that the Time_Wait state ports can be reused sooner.

For Linux, edit the file sysctl.conf by adding
net.ipv4.tcp_fin_timeout = 30
 

For Windows, please refer to this page: https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx

For Unix, please refer to this page: http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html

4. Update the Selenium jar and Webdriver binary files

The newer code always contain better features and more bug fixes. There could be some performance fixes or some changes to improve the functionality/performance for a specific browser. Meanwhile, the previous Selenium/webdrvier code may NOT be compatible with the latest up-to-date browser versions. So, check the Selenium HQ download page from time to time. When your browsers have a major release (for example, Chrome 44 to Chrome 45), you must check whether there are new Selenium jar or webdriver binary released. 

5. Retry!

All above good practices may help to improve the test stability from X% to 90%. However, they still can not ensure that the tests are 100% stable. The best strategy here is to retry the failed test cases. 

It is easier to do this if you are using Cucumber. Cucumber saves the failed test cases into a rerun file so that you can rerun them later. For other test framework, it is not too difficult also. As long as your script go through the test results for the first run, you should be able to pick up the failed ones and rerun them again. 

Summary

Selenium/Webdriver is a powerful web application test framework and is super popular. It is often the case that A Selenium grid is set up to support multiple concurrent test sessions. That way, the tests can finish quicker and can spread across different Browser/OS combinations. However, it is not easy to properly scale a Selenium Grid. That could be multiple moving parts and some architectural concerns involved. You may be able to apply a few good practices mentioned in this document to make your Selenium Grid more reliable and scalable. Although the 100% test reliability can not be achieved, we will achieve a good enough level, hopefully 90%. 

Last but not least, Selenium/Webdriver is great. Enjoy it.  

Install Ubuntu with Virtual PC on Windows 7

First of all, you can only use Ubuntu 9 with Virtual PC on Windows 7.

I tried Ubuntu 10 and used quite a lot tricks published on Internet. Unfortunately, none of them works.

Steps to Install Ubuntu 9 with Virtual PC

Step 1: Download the Ubutun 9.04 ISO

Do not use the Ubuntu 9.10 ISO. It will generate some issue related to missing applets.

Step 2: Create a Virtual PC machine

Set the D drive to point to the Ubuntu 9.04 ISO.

Step 3: Start the Virtual PC machine

Step 4: Follow the Ubuntu Install Wizard

If you run into any problem by using the Ubuntu default installed, you can use the following way

1. At the installation screen, use "English"
2. Then, press F6 to use the Startup Option. Replace "-- quite splash" with "-vga=791 noreplace-paravirt"
3. After getting into the main ui, click the icon "Install Ubuntu 9.04" on the desktop
4. Follow the wizard.

Step 5: Restart the Virtual PC Machine

The system will remind you to restart. Follow it. You can see a black screen. Do not be panic. Just click Enter.

Step 6: Login the Virtual PC Machine after it starts up

Step 7: Change the resolution to support 1024x768

You need to change the file /etc/X11/xorg.conf to following:

---------------------------------------------------------

# xorg.conf (X.Org X Window System server configuration file)
#
# This file was generated by dexconf, the Debian X Configuration tool, using
# values from the debconf database.
#
# Edit this file with caution, and see the xorg.conf manual page.
# (Type "man xorg.conf" at the shell prompt.)
#
# This file is automatically updated on xserver-xorg package upgrades *only*
# if it has not been modified since the last upgrade of the xserver-xorg
# package.
#
# Note that some configuration settings that could be done previously
# in this file, now are automatically configured by the server and settings
# here are ignored.
#
# If you have edited this file but would like it to be automatically updated
# again, run the following command:
# sudo dpkg-reconfigure -phigh xserver-xorg

Section "Device"
Identifier "Configured Video Device"
EndSection

Section "Monitor"
Identifier "Configured Monitor"
horizsync 30-60
vertrefresh 50-75
EndSection

Section "Screen"
Identifier "Default Screen"
Monitor "Configured Monitor"
Device "Configured Video Device"
DefaultDepth 16
SubSection "Display"
Modes "1024x768" "800x600"
EndSubSection
EndSection

---------------------------------------------------------

Step 8: Restart the Virtual PC Machine

Click the menu item "Restart" in Ubuntu 9. Do not directly turn off the VPC machine.

Option

You can upgrade 9.04 to 9.10. It should run smoothly. Meanwhile, the missing-applet issue does not happen this way.

Redshift Best Practice & Commands

Redshift Best Practices & Commands

Redshift Best Practices Presentations

http://www.slideshare.net/AmazonWebServices/amazon-redshift-best-practices

• Use S3 to store the processed data to be loaded. That way, the data can be retained and reloaded easily. 
• Use the proper Distribution Key
• If you have to do a few joins or group by clauses, it is better to use the fields in joins or group by clauses as the dist key. That way, all data to be loaded are stored in the same nodes. 
• Use the proper Sort Key. 
• It is good to use Timestamp field as a sort key if you have many queries based on the time. 
• Use two tables. One is the Staging table to load data and verify the loaded data. The other one is the Target table, aka Production table. That way, the data to be loaded can be verified and corrected. The product table will not be impacted. 
• Consider the retention policy and use UNLOAD command to archive unused data
• Unload: http://docs.aws.amazon.com/redshift/latest/dg/t_Unloading_tables.html
• Normalization rules:
• For non-collated joins, use de-normalization
• For slow changing dimensions, keep normalization, match the dist key with fact table
• Copy Command:
• Good for loading big data set
• Some good options: EMPTYASNULL, BLANKASNULL, GZIP
• DateTimeFormat: Must have a space between the date part and time part.
• Space for Load and Vacuum. 
• Load and Vacuum needs 2.5 time of the table to operate. 

Useful Commands

Space Monitoring

The command to analyze space for a cluster
select
    trim(pgdb.datname) as Database,
    trim(pgn.nspname) as Schema,
    trim(a.name) as Table,
    b.mbytes,
    a.rows
from (
    select db_id, id, name, sum(rows) as rows
    from stv_tbl_perm a
    group by db_id, id, name
) as a
join pg_class as pgc on pgc.oid = a.id
join pg_namespace as pgn on pgn.oid = pgc.relnamespace
join pg_database as pgdb on pgdb.oid = a.db_id
join (
    select tbl, count(*) as mbytes
    from stv_blocklist
    group by tbl
) b on a.id = b.tbl
order by mbytes desc, a.db_id, a.name;


Analyze Table distribution between nodes:
select slice, col, num_values, minvalue, maxvalue
from svv_diskusage
where name = '__INSERT__TABLE__NAME__HERE__' and col = 0
order by slice, col;

The command to analyze compression

Data Loading

Bugs found in testing a Hybird Mobile Application

1. The javascript Date.parse does not work well with Android 2.3
What: Data.parse function does not parse the Unix format timestamp well
Why: Has to reformat the timestamp, then parse the date

Selenium Server 运行 HTMLSuite Selenese

You can run the HTML Suite Selenese script with the selenium-server.jar
The reference document is: http://seleniumhq.org/docs/05_selenium_rc.html
The sample command is:
java -jar selenium-server.jar -htmlSuite "*firefox" "http://www.google.com" "c:\absolute\path\to\my\HTMLSuite.html" "c:\absolute\path\to\my\results.html"

The real command executed by me is:
E:\Projects\Selenium\Google>java -jar C:\selenium-remote-control-1.0.3\selenium-server-1.0.3\selenium-server.jar -htmlSuite *firefox http://aws.amazon.com E:\Pr
ojects\Selenium\Google\AWS_Signin_Suite.html E:\Projects\Selenium\Google\AWS_Sig
nin_Suite_Report.html

Notes:
1. The selenium works as a proxy. It will retrieve all the embedded resources in a request on behalf of the browser. The selenium is much much slower while processing resources. So, you may need to set the timeout to 600 seconds
2. *chrome can be used to launch *firefox browser also.
3. -log option can be used to track the server side activities.
4. Sometimes, you may see "Socket: Read time out" exceptions. It is normal and does not mean the test case fails. Just be patient.

Selenium CSS Locator Tips

How to use nth-child?
nth-child can not return you the nth element in a matched set. It only return the elements who are the nth child of their parent nodes.

Why the Descendant selector does not work with nth-child?
It may be a Selenium bug.
This css locator will not work
css=div.application_details tr:nth-child(1) span.label
But the following one works
css=div.application_details tr:nth-child(1)>td span.label

It seems that Selenium gets confused after nth-child is used. We can use a direct child element to make the Selenium back to sane again. :-)

Top 10 Performance Techniques for PhoneGap Applications

by Christophe Coenraets on May 13, 2013 in PhoneGap

http://coenraets.org/blog/2013/05/top-10-performance-techniques-for-phonegap-applications/

1. TEST:  Test with the actual data and devices to feel the performance
2. Avoid reflow: Modify the DOM once. Put all data together, then update the DOM. Not frequently update the DOM. 
3. Do you really need the framework: Or you can just apply some CSS tricks to archive the same goal
4. Limit shadows and gradients:
5. Use CSS sprite sheets: Do not load each image individually
6. Avoid click event 300ms delay: Use touch event
7. Use hardware acceleration: Use CSS  transform: translate3d(-100%,0,0);  It will use GPU, instead of CPU.
8. Cache everything: cache static content, cache templates
9. Don't wait for the data to display the UI: display the UI, and update it when the data is ready. People do not wait on the UI, they are waiting for the data. 
10. Don't generate the UI on the server