Showing posts with label OTD. Show all posts
Showing posts with label OTD. Show all posts

Tuesday, 7 July 2015

Oracle Traffic Director - Deployment options, Virtual Servers vs Configurations

Summary

Oracle Traffic Director (OTD) is a powerful software load balancing solution.  As with most good products there is a degree of flexibilty in how it can be deployed with different approaches allowing the solution to be formed.  This article discusses two options that could be used to determine different routing possiblilties.

The scenario that is being considered is a need to perform two separate load balancing activities in the same OTD environment.  For example, load balancing to an older SOA 11g deployment and to SOA 12c for recent integration deployments.  Another possible example would be two routes to the same back end service but one is designed for high priority traffic while the other route will throttle the service at a preset load.   The two options that are discussed are:
  1. Using two separate configurations, one for SOA 11g and one for SOA12c.
  2. Using one configuration that has two virtual servers.  The virtual servers handling the routing for each environment.
Needless to say either option can be appropriate and it will depend on the details of the overall solution and to some extent personal preference to determine the right answer for a particular customer environment.  Of course other options such as more complex routing rules within a single configuration or multiple OTD domains are also options to think about.

OTD Configuration Overview

Simple configuration

An OTD deployment, in its simplest form, consists of an administrative instance which manages the configuration and a deployed instance.  The deployed configuration specifies the HTTP(S)/TCP listening port, routing rules to one or more origin servers, logging setup etc.   In many situations there is a business need to use OTD to manage requests to different business applications or even just to different environments/versions of an application.  It is obviously possible to split these out by using independent deployments of OTD however to minimise the resources required and keep the number of deployed components to a minimum there are options to use one administration server.

The base configuration options


The minimum configuration that will appear for a configuration is a setup which defines things like the listening ports, SSL certificates, logging setup and critically at least one origin server pool and a virtual server.  The origin server pool is a simple enough concept in that it defines the back end services to actually fulfil the client requests. 

Using Virtual Servers

The virtual servers provide a mechanism to isolate traffic sent to the software load balancer.  Each virtual server contains its own set of routing rules which can determine the origin servers to send requests to, caching rules, traffic shaping and overrides for logging and the layer 7 firewall rules.  The virtual server to be used for subsequent processing is identified by either the listening port or the hostname used to send the request.

Virtual Server example - Routing based on otrade-host
Virtual Server example - Routing based on websession-host
So in the above example both hostnames otrade-host and websession-host resolve to the same IP address in DNS (or in the clients local /etc/hosts file).   In this case two virtual servers also use the same listener.  If the client makes a request to access otrade-host then the first virtual server is used and if they request websession-host then the second's rules are used.

There is always at least one virtual server.  By default this is created and the hosts field left blank such that it is used if any traffic hits the listening port.

Solution Variations

Multiple Configurations

Overview

In this setup two configurations can be defined and deployed.  It is quite possible to have both configurations deployed to the same OS instances. (Admin node in OTD talk.)  The result of deploying the configuration to the admin nodes is the creation of another running instance of OTD.
Running multiple configurations
Thus in the example shown above we have three OS instances, one to host the admin server which could be co-located with the actual instances.  There are two OS instances which host two OTD servers, one for each configuration.  I have shown two OS instances to run the configuration to indicate that they can be setup in a failover group to provide HA, each config can utilise a different VIP.

Advantages

  • Each configuration is managed independently of each other.  (within the one administration server) 
    • The settings are independent of each other.
    • The running instances for each configuration are independent of each other. i.e. Can be stopped and started without impacting the other configuration instances running.
  • Simple to understand

Disadvantages

  • Care must be taken to ensure that the configurations do not have clashes with each other.  (eg. Same listenting ports)
  • Results in more processes running on each OS instance.

Multiple Virtual Servers

Overview

In this situation there is one configuration with multiple virtual servers which result in different routing rules being applied to send requests on.   In the diagram below we have deployed a single configuration to two OS instances with the configuration containing two virtual servers.  As per the multi-config option I have shown two OS instances to indicate that the failover group can be used for HA.

OTD Using Two Virtual Servers

Advantages

  • One configuration that provides visibility of all configuration in the environment.
  • Minimal running processes
    • Simplifying the monitoring
    • Reducing resources required to run the system

Disadvantages

  • Introduces dependencies between the environments
    • eg. Can share listeners, origin server pools, logging config etc.  Thus one change can impact all instances
    • eg. Some changes mandate a restart of an instance.  A change for one config may have an impact on load balancing for the other environment.
  • Complexity of a single configuration
  • Dependencies on external factors.  (DNS resolution of hostnames/firewalls for port access.)

Conclusions

There are no hard and fast rules to figure out which approach is the best one for you.  It will ultimately depend on the requirements for the load balancing.  If a configuration is changing frequently and is functionally independent then I would tend to go for the multiple configuration route.  If on the other hand simplicity of monitoring and minimal resource footprint alongside a fairly static configuration was the situation I would tend to use the multiple virtual server approach.

Essentially the classic IT answer of "it depends" will apply.  Only a good understanding of the requirements will clarify which way to go.  (Although if you are using OTD 11.1.1.6 then you might be better with the virtual server approach as there are a few limitations to the VIPs using keepalive for the failover groups)

Friday, 13 June 2014

Example of Application Version Management Using OTD

When using Exalogic to host an application it is fairly common to see Oracle Traffic Director (OTD) being used as a load balancer in front of the application.  OTD has a rich functional set that can make it more than just a load balancer/proxy - this allows it to perform traffic management/shaping.  In a recent engagement I was asked if it could perform application version management, to which the answer is yes.

The solution presented here is just one possible answer to application versioning and there are often many other approaches that should be considered.

Problem Statement

The scenario we are looking at here is a situation where a new JEE application is being deployed to an application server and the new version is to be set live on a particular date/time.  Existing sessions with the old application are to carry on using the previous version of the application until their sessions expire.
There are a number of restrictions that apply:-
  • No changes to the underlying application.  (For example, WLS has the concept of versioning built into it but the application code - descriptors - must be altered to use this.  In our scenario we are using a third party application and do not have the ability to make any changes.)
  • No application down time is allowed.  (App is accessed 24*7)
  • Existing sessions remain on the previous version and new sessions use the new version

Solution Overview

The solution used makes use of the routing rules that are available in OTD.  Essentially inspecting the HTTP requests that hit OTD and determine if the request should remain on the old application version or be routed to the new version.

The first problem is to figure out how to know if a request is to be routed to the old or new version of the application.  Assuming the back end application is an application server it is very likely that a jsessionid cookie will be set for any existing client connection.  If not set then we could route what will be a new session onto the new application version.  The problem comes with the next request, it will have the jsessionid cookie set and we have no way of knowing if the session is to be routed to the old or the new application version.  A solution to this issue is to use OTD to inject an additional cookie into the browser, this cookie can contain the time that it was created.  Then we can simply inspect this cookie on each request and if the time it holds is before a cut-off date then route to the old version and if after the cut off then route to the new application.

Solution Detail

Injecting a Cookie

The first activity is to use OTD to inject a cookie into the HTTP response that goes back to the client.  This can be done using the Server Application Functions (SAFs) that OTD supports, these are documented in this link.  (Although check for newer documentation releases from here.)

In summary OTD makes use of a special file the obj.conf file that contains a list of directives to instruct OTD how to handle incoming HTTP requests.  Many of the changes done through the OTD administration console or command line end up as directives in this file.  In addition to the commands that are neatly wrapped up in the console interfaces there are additional directives and functions that can be added by editing the correct obj.conf file by hand.  Inserting a cookie is one such SAF.

The first activity is to understand which file to edit.  If you simply do a search from the command line under the OTD instances directory for all *.obj.conf files you will see that there are multiple copies of files held.  This is because each configuration/virtual server has its own copy of the file and these are held centrally in a repository and copied out to the running instances when changes are deployed.  In our scenario we are wanting to change the HTTP processing for a particular "Virtual Server" as part of an OTD configuration.  The path to the file we are interested in is:-



<OTD Instance Home>/admin-server/config-store/<configuration name>/config/<virtual 
server name>-obj.conf



This resides on the server that is hosting the OTD administration server which manages all the obj.conf files for all the configurations/virtual servers used.  The <virt-server>-obj.conf file is the copy held in the repository, it can be edited to add a section that will inject the cookie.  Having edited the file then either use the command line (tadm) or the browser to deploy the configuration file.  The action of deploying the config file will copy the configuration to the running instances and update them so that they use the new configuration.  This update is done dynamically and does not require any downtime of the OTD instance.

A sample directive to add the cookie is shown below:-



<Object name="default">
AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"
NameTrans fn="assign-name" id="default-route" name="default-route"
NameTrans fn="map" from="/" to="/"
<If not defined $cookie{"MyApp"}>
ObjectType fn="set-cookie" name="MyApp" value="$time"
</If>

Service fn="proxy-retrieve" method="*"
AddLog fn="flex-log"
</Object>

<Object name="default-route">
...



So in this case if the cookie has not been presented to OTD by the client browser then OTD will add a cookie called "MyApp" with a value of the current time.  (In seconds since the epoch).  The client will then present this cookie in all subsequent requests.  It is possible to set additional options such as the domain and path which can specify when the client will send the cookie or to set a max-age parameter which will make the cookie persistent with the browser and last for the duration specified, in seconds.  In our example the max-age is not set so the cookie will last for the duration of the session.  (until the browser is shutdown)

Handy Hint
With unix it is fairly easy to get the current time in seconds since the epoch using the # date +%s command which will simply format the current date.  Or to convert from a number of seconds to a meaningful date use # date -d @1402655677

Setting up a Routing Rule

Having added the cookie we now have something that we can easily use to route with.  This can be done either by editing the obj.conf file directly or using the browser interface which provides some help in ensuring that the syntax is correct.   To add/edit routing rules simple ensure that you have selected the configuration you want to manage and then open up the virtual server in the navigation pane and it is possible to then add the routing rules.

In our scenario we are going to add a routing rule that will inspect the "MyApp" cookie and if the value is greater than a set time then route the request to the next version of the application.  This could be as simple as routing to a different origin server pool if the new application has been setup in an entirely different set of WLS instances, or as in this case the rule will cause a re-write of the request to access a different URL where the two applications (new and old) are both held in the same application server but with different base URLs. 

Routing rules to direct requests to the new app version


These rules can then be deployed to the server instances which again is done dynamically with no down time.  Then when the clients access the service if not present they will have the "MyApp" cookie added to their browser and each request is then checked against the cookie value and if the value is greater than the time specified in the rule then the request will be routed to the second version of the application.  Sessions that started on before the cutoff time will continue to use the previous version of the application using the default rules.  (or other routing rules.)

The changes to the routing rules are reflected in the obj.conf file as shown below.


#
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
#

# You can edit this file, but comments and formatting changes
# might be lost when you use the administration GUI or CLI.

<Object name="default">
AuthTrans fn="match-browser" browser="*MSIE*" ssl-unclean-shutdown="true"
<If defined $cookie{"MyApp"} and $cookie{"MyApp"} gt "1402483929">
NameTrans fn="assign-name" id="myapp-v2" name="myapp-v2"
</If>

NameTrans fn="assign-name" id="default-route" name="default-route"
NameTrans fn="map" from="/" to="/"
<If not defined $cookie{"
MyApp"}>
ObjectType fn="set-cookie" name="
MyApp" value="$time"
</If>
Service fn="proxy-retrieve" method="*"
AddLog fn="flex-log"
</Object>

<Object name="default-route">
Route fn="set-origin-server" origin-server-pool="origin-server-pool-1"
</Object>

<Object name="myapp-v2">
NameTrans fn="map" to="/MyApp-v2" from="/MyApp"
Route fn="set-origin-server" origin-server-pool="origin-server-pool-1"
</Object>




Obviously once the previous application version has been fully retired then the default rule can be used to direct to the new version and this routing rule deleted.  Potentially the cookie injection can also be removed.

Conclusion

With a very simple configuration consisting of a single cookie injection and a routing rule it is possible to use Traffic Director to manage different application versions. 

Tuesday, 11 February 2014

Websocket support in Oracle Traffic Director

Introduction

In recent releases the Websockets function has been added as a capability to both WebLogic 12.1.2 and Oracle Traffic Director (11.1.1.7.0).  This blog posting is another courtesy of my colleague Mark Mundy who has been doing some investigation into what is involved to setup an architecture with a browser accessing a web application running websockets on WebLogic and proxied/load balanced through OTD.

The WebLogic documentation covers the setup of Websockets at the server side and the OTD documentation explains what is needed for the load balancer.  This blog posting is intended to show how the two can be used together on the latest Exalogic virtualised release to have OTD load balance a simple example websocket application with relative ease. This posting will not go through every aspect of the setup as the assumption is that the reader has some familiarity with Exalogic Control, WebLogic and OTD.  The example application being used is the one that ships as part of the example set for WebLogic 12.1.2 with a slight modification as will be highlighted.
The deployment topology that is used in this scenario is shown below.
WebSocket deployment Topology - Browser --> OTD Load Balancer --> Application(WLS Cluster)

Setting up the Exalogic Environment 

Depending on how complex you want to make the environment, will determine what needs to be set up. In order to demonstrate the basic load balancing while ensuring that there is no direct network connectivity between the client browser and the back end WLS managed server hosting the websocket application, a minimum of 2 vServers would be needed. In order to test out features such as load balancing during failure scenarios at both WLS and OTD ideally 5 would be created. For this example it will be assuming 5 vServers are created and the topology reflected in the following diagram shows the networking setup.
 
Exalogic vServer deployment topology

WebLogic Installation and Configuration

WebLogic 12.1.2 is the minimum version that can be used for Websocket support, select the 12c Generic Download from the Oracle download site.  (This will have a dependency on the latest 64bit java version so if not already installed on your vServers then install prior to installing WLS.)
Install the Complete Installation of WLS 12.1.2 using the installed 64 bit Java 7.  Installing the complete installation will ensure the example applications are available and it is the Websockets example that will be used.  Once WLS is installed complete the post installation configuration to ensure this happens. For more details on getting the examples available see this link
The specific websockets example application used in this document will be introduced after a specific WLS domain is created to host it.
Rather than use the WLS examples domain a new domain will be used so as to keep things as simple as possible but at the same time allowing the ability to have more flexibility in the testing. Further detail about the WLS implementation is available in the documentation.

A new WLS domain needs to be created using the domain wizard.  In the topology shown above the two instances in a cluster that listen on the private vNet and the administration server uses this network as the default but also has an additional HTTP channel setup to listen on the EoIB public network so that administration becomes simple from outside the Exalogic rack.  The websocket application can then be deployed to the cluster such that is is only accessible to the client population through the OTD tier, providing a level of security.

The example that ships with WebLogic will only work in the scenario where the browser and WLS managed server are both on the same host. To resolve this issue we make a small change to the code in the example before building the application and deploying to the cluster.  In the example source code identify exampleScript.js and change the line that opens up the websocket.
This Javascript file is normally located in:-
<WLS_Home>/user_projects/applications/wl_server/examples/src/examples/webapp/html5/webSocket/src/main/webapp/script
Use your editor of choice and search for "localhost:7001" and replace it with " + window.location.host +" as shown below.

...
if (ws == null) {
      ws = new WebSocket("ws://localhost:7001/webSocket/ws");
      ws.onopen = function() {
        addMsg("Server Msg: WebSocket has been started.");
      };

...
becomes
...
if (ws == null) {
      //ws = new WebSocket("ws://localhost:7001/webSocket/ws");
      ws = new WebSocket('ws://' + window.location.host + '/webSocket/ws');

      ws.onopen = function() {
        addMsg("Server Msg: WebSocket has been started.");
      };

...

The directory <WLS_Home>/user_projects/applications/wl_server/examples/src/examples/webapp/html5/webSocket contains full instructions on building and deploying the application.  In our case we only need to re-build the application (ant build) and then use the standard WLS admin console (or WLST or ant if you prefer) to deploy the war file created to the WebLogic Cluster.

Oracle Traffic Director Installation and Configuration

For Websocket support we need to ensure that we install OTD 11.1.1.7.0 or higher.  In our scenario we will install OTD onto the admin vServer and the two OTD instance vServers to end up with an HA cluster hosted on the two instance vServers that will route requests through to the backend WLS cluster.  It is useful to enable monitoring on OTD so that we can easily spot the sessions being created in OTD and through to the back end WLS instances.  To enable monitoring highlight the virtual server and enable the plain text report.  As shown below.

Enable plain text monitoring in OTD

Websocket support is enabled by default in OTD so there is nothing else other than the deployment of the configuration to the cluster before we test out our Websocket application.  However for information if there is a desire to disable websockets for a particular route then this can be achieved via the enable/disable of the "WebSocket Upgrade" for a given route.


Enable/Disable Websocket support in OTD.

With the monitoring in place it is then possible to access the plain text report by accessing the URI http://OTD IP address:port/.perf where the Origin Server statistics are of particular interest.  As highlighted below.


Testing the application

In another browser session/tab it is then possible to access the OTD instance with the URN /websocket.  You can then click the start button to open up the socket and enter text to send to the server.  Currently you should use either Chrome or Firefox but not Internet Explorer as it does not support websockets.


WLS WebSocket example application

By starting up additional browser sessions you can create multiple websockets and the .perf output will reflect the load balancing to the back end WLS instances.


In this example we can see that 3 websocket sessions have been created and they have been load balanced round the two available WLS server instances.

Behind the scenes what is actually happening is that there are two socket connections created for each WebSocket.  The first is between the client browser and the OTD instance and the second from OTD to the WLS instances.  OTD acts to bridge the gaps between the two.  Below is the output from netstat on the OTD instance to show these socket connections.

Browser to OTD instance.
# netstat -an
Active Connections
Proto  Local Address          Foreign Address        State
TCP    138.3.32.68:52732      138.3.49.9:8080        ESTABLISHED


& OTD to WLS
# netstat -an | grep 7003
tcp        0      0 192.168.16.74:39223         192.168.16.71:7003          ESTABLISHED


The nature of establishing multiple sockets to link the load balancer to the back end service means that in the event of a failure by either OTD or WLS then the socket connection will be broken and the client must re-establish the socket again.  Thus for HA connections there is a dependency on the client capturing any exceptions and re-establishing the connection.
Note:- If OTD is setup with a failover group (HA config) then one instance is primary and the second acts as a backup.  Should the primary fail then the websocket would have to be re-established using the backup OTD instance.  Once the primary is available again OTD will automatically fail back which will again break the websocket connection.