Visualizing the Enterprise:

George Belotsky

Open Light Software Inc.

questions@openlight.com

Left-click for next slide. Move the mouse cursor to the bottom right of the page; additional controls will appear. This slide is based on the S5 presentation system.

What We'll Be Covering

• How to automatically generate graphs, charts and other diagrams for your web application.
• Which Open Source tools you can use
  • To get you started quickly, concise examples of just a few slides each will illustrate the tools.
• Design patterns for web-based visualization, to help you build your own solution.

High-Level Choices

• Generate Visuals on the Server.
  • Create images (such as PNG or GIF) that the web browser can load and display.
  • Can also produce PDF files for printing.
• Generate Visuals on the Client (Browser).
  • Using JavaScript, Java Applets or Flash.
  • In this talk, all "clients" are always browser-based.
• Hybrid Approaches.
  • Use a server-generated image as the background for when rendering graphics on the client. Example 4 illustrates this.

Server-Based vs Client-Based

Server-Based Tools

• JFreeChart (creates a variety of graphs and charts).
• GD (low-level library).
• Graphviz ("An automated Visio").

Client-Based Tools

• JavaScript with the wz_jsgraphics library.
• Flash with the MTASC compiler.
• We will not discuss Java applets, since browser support is less uniform than before. This can still be a good solution, however, especially on an intranet.

JFreeChart

Example 1: Survey Results using JFreeChart

Here is a simulated survey, illustrated using JFreeChart. Try It

Source Code for Example 1

1. Initialize. Remember to run the program with the -Djava.awt.headless=true option if X is not installed on your server.

  import java.io.*;
  import java.awt.image.BufferedImage;

  import org.jfree.chart.*;
  import org.jfree.data.general.DefaultPieDataset;

      ...

Source Code for Example 1 -- Cont.

2. Insert the data.

    ...

  DefaultPieDataset data = new DefaultPieDataset();
  for (int ii = 0; ii < values.length; ++ii) {
      data.setValue(labels[ii], values[ii]); 
  }	

Source Code for Example 1 -- Cont.

  JFreeChart chart = ChartFactory.createPieChart3D
      (new String(name), data, true, true, true);
	
  //Make the chart translucent.	
  chart.getPlot().setForegroundAlpha(this._opacity);
	
  BufferedImage buffer = chart.createBufferedImage(this._width, 
                                                   this._height);

  return ChartUtilities.encodeAsPNG(buffer);
  //ChartUtilities.writeBufferedImageAsPNG(ostream, buffer);
  //ChartUtilities.saveChartAsPNG(new File("survey.png"), chart, 
  //                              this._width, this._height);
    ...

GD

Example 2: Item Finder using GD

Suppose you want to show the location of various items (products in a warehouse, trucks on the road, etc.) on a map. This is easy to implement with GD. Here is an example, written in Python. Try It

Source Code for Example 2

1. Initialize.

  import gd
  import cStringIO

  image = gd.image('map.png')

  tag_color = image.colorAllocate((255, 0, 0))
  txt_color = image.colorAllocate((0, 0, 0))

Source Code for Example 2 -- Cont.

2. Mark item's location with a circular pattern.

  if (itemname):
      for ii in range(30, 5, -5):
          image.arc((item_x, item_y),
                    (ii, ii), 0, 360, tag_color)

      image.fillToBorder((item_x, item_y),
                         tag_color, tag_color)
        
      image.string(gd.gdFontGiant, (0, 0),
                   'Location of '+itemname, txt_color)

Source Code for Example 2 -- Cont.

3. Generate the generated image in memory, and return it.

 capture = cStringIO.StringIO()
    try:
        image.writeGif(capture)
        return ("image/gif", capture.getvalue())
    finally:
        capture.close()

Graphviz

• Graphviz draws "node-and-edge" diagrams (i.e. graphs in graph theory) similar to Visio, except that Graphviz is automated.
• The package provides lots of features to customize the look of the generated graph.
• Graphviz can generate graphs with hundreds of nodes -- much more than is practical manually on a regular basis.
• There are Graphviz bindings for many languages (including Python, Perl, PHP, Ruby and Java).

With Graphviz, you can regularly regenerate "Visio-style" diagrams (including very large ones) to reflect current data.

Example 3: Orgchart Using Graphviz

Here is an organizational chart generator, using Graphviz. This is a Python program, which utilizes the Python bindings supplied by Graphviz 2.8. Try It

Source Code for Example 3

    import os
    import time
    import sys

    #Replace with your own location, or append to PYTHONPATH
    sys.path.append('/usr/local/lib/graphviz/python')
    import gv

    g = gv.graph('Orgchart')

    #Set default node properties
    proto = gv.protonode(g)
    gv.setv(proto, 'shape', 'box')
    gv.setv(proto, 'fillcolor', 'blue')
    gv.setv(proto, 'style', 'filled')
    gv.setv(proto, 'fontcolor', 'white')

Source Code for Example 3 -- Cont.

2. Specify all superior subordinate relationships. Set employee titles, and mark all the important people.

   for boss, underling in bossof:
       rel = gv.edge(g, boss, underling)

   for someone in important:
       gv.setv(gv.findnode(g, someone), 'fillcolor', 'red')

   for lord in title:
       gv.setv(gv.findnode(g, lord), 'label', 
               lord+'\n('+title[lord]+')')

Source Code for Example 3 -- Cont.

3. Specify the layout, then render the graph. Currently, the Graphviz Python bindings will only render to a named file. As a workaround, you can create temporary filenames (make sure to do so securely, and to clean up the file after you are done). There are also alternative Python bindings, which are listed on the Graphviz website.

   gv.layout(g, 'dot')
   gv.render(g, 'gif', 'orgchart.gif')

Integrating the Server-Based Tools

• The simplest case is when the tool comes as a native library for your chosen programming language.
• If not, then software such as SWIG can create cross-language bindings. For popular Open Source tools, someone has probably already done this work.
• If the tool is a standalone program, it may be possible to control it from another process -- the classic scripting solution.

More Scripting Options

• If you choose the scripting approach, you can readily write a small wrapper to make a library-based tool standalone.
• Sometimes, tools have both library and standalone versions already (e.g. Graphviz). If not, someone may have already written a wrapper (e.g. fly is such a command-line wrapper for GD).
• When scripting, you typically launch the tool (in a separate process) every time you use it. Some tools, however, can be kept alive and reused. This allows you to create a process pool for handling requests, which can be much more efficient.

Turn Dynamic Content Into Static Content

• Dynamic web content has evolved since the CGI days.
• Still, scalability and security remain serious problems.
• Whenever possible, convert dynamic content into static content.

The "Relative Static" Solution

• Write a program unrelated to the web server. The program will gather data, process it and run the graphics tools to create the image file.
• The program can utilize a daemon process to generate the image periodically or based on some event. Alternatively, someone can manually run the program when new data is available.
• The newly generated image file replaces the old one that the web server is using.

On to Client-Based Tools

• wz_jsgraphics (with JavaScript).
  • We will also use the prototype library (see slide 28) in this example.
• MTASC (with Flash)
  • Note that the Flash plugin itself is not Open Source, but the MTASC Open Source ActionScript compiler is all you need to create sophisticated Flash-based clients.

wz_jsgraphics

• Pure JavaScript low-level graphics library.
• Provides 2D rendering facilities (lines, ellipses, polygons, etc.).
• This library allows you to write interactive Web 2.0 graphical applications using Ajax.

With wz_jsgraphics you can develop cross-browser, interactive graphical applications now.

Example 4: ItemFinder with wz_jsgraphics and Ajax

This is a variation on example 2. The new code uses an image from the server, but draws the item marker on the client, using wz_jsgraphics. Enter items "cubevan11" or "flatbed28" in the demo's "Item" textbox. Try It

Example 4: Supporting Tools

• The prototype library (http://prototype.conio.net/).
• Lightweight alternative: prototype.lite.js and moo.ajax.js (http://moofx.mad4milk.net/documentation/). Note that the Ajax syntax is slightly different with moo.ajax.js.
• The JavaScript compressor from the Dojo toolkit (http://dojotoolkit.org/docs/compressor_system.html). This tool can significantly reduce the size of your scripts.

Source Code for Example 4

1a. The ItemFinder class, constructor.

var ItemFinder = Class.create();

ItemFinder.prototype = {
    initialize: function(url, markf) {
        this._url = url;
        this._markf = markf;
    },

    ...

Source Code for Example 4 -- Cont.

    ...
    getitem: function(name) {
        markf = this._markf;

        new Ajax.Request(this._url+name, {
	    method: 'get',
	    onSuccess: function(r) {
		coords = r.responseText.split(',',2);
		markf(Number(coords[0]), Number(coords[1]));
	    },

	    onFailure: function(e) {
              alert('Error (make sure item exists)');
	    }});
    }   
}

Source Code for Example 4 -- Cont.

2. Load the necessary libraries Note the <div> at the bottom. The drawing will take place there.

  ...
<script src="/s/prototype.js" type="text/javascript"></script>
<script src="/s/wz_jsgraphics.js" type="text/javascript"></script>
<script src="/s/itemfinder.js" type="text/javascript"></script>
</head>

<body>

<div id="map">
<img src="map.png" alt="Item Location" 
width="283" height="320">
</div>
  ...

Source Code for Example 4 -- Cont.

3. The controls which send user-generated events to the application.

  ...
<div>
<label>Item: </label><input id="item" type="text" size="15"/>
<button onclick="finder.getitem($F('item'))">Get Location</button>
</div>
  ...

Source Code for Example 4 -- Cont.

  ...
<script type="text/javascript">
   var canvas = new jsGraphics("map");
   canvas.setColor("#00ff00");
   canvas.setStroke(2);  
   
   var finder = new ItemFinder('items/', marksquares);
   
   function marksquares(x, y)  {
     canvas.clear();
     for (var ii = 32; ii > 0; ii -= 8) {
       canvas.drawRect(x-ii/2, y-ii/2, ii, ii);
     }
     canvas.paint();
   }
</script>
 ...

Client-Based Designs

• You can use XML, JSON, etc. as the foundation of a network protocol, but these tools are not a substitute for the protocol design work that your application requires.
• A client can be specialized to render specific data (e.g. mark a map with a circle) or generalized (e.g. draw any shape that the server requests).
• Bandwidth and security are important. Note that you can use compression and encryption as part of your protocol to help with these issues.

Generic Clients vs Specialized Clients

Limiting the Client

• The server should be able to identify client version, and display an error message to out-of-date clients.
• Clients that poll the server periodically should stop polling on server request (cooperative bandwidth control).
• Clients should self-limit their maximum polling rate; or even stop altogether if unattended for long periods.
• Sessions should expire. For additional safety, both the client and the server can include this functionality.
• Depending on your environment, you may still need anti-abuse controls on the server, such as refusing requests from IP addresses which connect too often.

MTASC

• MTASC is a fast, command-line ActionScript compiler.
• All standard Flash API features are accessible.
• Can be used with Macromedia (now Adobe) tools, or by itself.

Flash is not just for dumb movies -- it is an advanced programming environment. Flash-based clients can be very sophisticated, and browser support is excellent.

Compiling a Flash Client

Replace all paths to where MTASC resides on your system. The header option creates a Flash "movie" from scratch. Specify the file where your main resides; the compiler will find the other source code files automatically.

/opt/bin/mtasc -cp /opt/lib/mtasc/ -version 6 -strict \
 -swf canvas.swf -header 800:600:20 -main FdUI.as

Example 5: Monitoring Application Using Flash

We finish with a more complex example, which embodies a lot of the concepts covered in this talk. Here is a complete Open Source monitoring application (written by the presenter). You can download all the source code and view the demo at http://www.openlight.com/fdui Try It

Source Code for Example 5a -- Rendering

 class Canvas {      
    ...
 public function draw_poly(moviename:String, points:Array,
                           color:Number):Void {
        var movie:MovieClip = this.parent[moviename];
        with(movie) {
            beginFill(color,100);
            lineStyle(0,color,100);
            moveTo(points[0][0],points[0][1]);
            for (var ii:Number = 1; ii < points.length; ++ii) {
                lineTo(points[ii][0],points[ii][1]);
            }
            lineTo(points[0][0],points[0][1]);
            endFill();
        }
    }
    ...

Source Code for Example 5a -- Cont.

You can create movie clips as your ActionScript program runs. Movie clips layer on top of one another. If you set one clip to be partially transparent, the one underneath will show through.

 class Canvas {     
       ...
    public function clear(moviename:String):Void {
        this.parent.createEmptyMovieClip(moviename,
                                         Number(this.names[moviename]));
    }
       ...
  }

Basic ActionScript Networking

• For simple name-value pairs, use a LoadVars object. (use an XML object for XML-based protocols).
• Set the object's onLoad member to a callback function, to handle load events.
• Use the object's load, send and sendAndLoad methods to communicate with the server.

Source Code for Example 5b -- Network Protocol

ActionScript supports interfaces, similar to Java. Here is the Protocol interface from Flightdeck-UI Online. Making the protocol replaceable is very important for extensibility, and greatly eases security patching.

   interface Protocol {
     public function init(hostdata:LoadVars, id:String):Void;
     public function input(hostdata:LoadVars):Object;
     public function output(hostdata:LoadVars):Void;
     public function remote_log(msg:String, level:String):Void;
   }

Source Code for Example 5b -- Cont.

Even with client-based visualization, the server usually supplies the underlying data. Hence, the "Relative Static" approach also works here. Flightdeck-UI Online uses a separate daemon process to periodically regenerate the data, and save it to a file.

$ cat demo.txt
reset\needles\2|poly\needles\697.0\501.5\691.0\498.5\ ...

Here, you can see the server's instructions to draw polygons into the "needles" movie clip of the Flash client. This updates the gauges.

Source Code for Example 5b -- Cont.

Protocol version check.

class Proto_X1 implements Protocol{
  ...
 public function input(hostdata:LoadVars):Object {
	if (!Number(hostdata.proto_x1)) {
	    this.remote_log("Wrong protocol version; expected x1")
	    return null;
  ...

Source Code for Example 5c -- Limiting the Client

In addition to the dedicated data generator daemon, Flightdeck-UI Online uses Apache with a mod_python script to control the client. The code below shuts off outdated clients.

def handler(req):
       ...
   ver = fields.getfirst('ver','').strip()
       ...
   if not versions.has_key(ver): #!!! Length-limit this?
      req.write('command='+Client.shutoff_panel('Version Expired ...
       ...

Source Code for Example 5c -- Cont.

Of course, the client must allow the server to control it. During initialization, the Flightdeck-UI Online client registers a "stop" command ("stop_cmd", below) for this purpose.

public function FdUI() {
       ...
   var protocol:Protocol = new Proto_X1({test:[this,"test_cmd"],
				       rclr:[this,"regclr_cmd"],
				       rsnd:[this,"sndreg_cmd"],
				       snd:[this,"sndplay_cmd"],
				       arc:[this,"arc_cmd"],
				       poly:[this,"poly_cmd"],
				       label:[this,"label_cmd"],
		                               ...
				       stop:[this,"stop_cmd"]},
				       this.version,
				       this.max_command_len);
       ...

Conclusion

• Presenting graphical data over the Web is an efficient method to disseminate information to customers, coworkers, or the general public.
• There are lots different Open Source tools that you can use in your solution. Many of these tools work very well together.
• A variety of designs are applicable to web-based visualization. For example, the "Relative Static" approach scales very well, and will handle most typical situations