Events over time chart using Flot Charts

At times we have a time series data and would want to plot the count of events over time. This post is to show how we can do it with Flot charts.

The output

Image

The data we have at hand is just a list of timestamps. Let’s take the twitter stream as an example. I extracted the “created_at” timestamp from every twitter event and formed a List<String> in java. Now I want to plot this as events over time.

Flot charts expects you to send an array of co-ordinates like [ [x1,y1], [x2,y2], … ]. Here x has to be the unix timestamps in millis and y has to be the count. This leaves us to compute y as we wish.

The algorithm I used for computing y is as follows:

1. Find diff = max Timestamp (X max) – min Timestamp (X min)

2. If the diff is less than 5 mins, count events per second.

or, if diff less than 5 hours, count events per min

or, if diff less than 5 days, count events per hour

or, else count events per day.

I did this so that the count window is dynamic to make the chart look good. The number 5 can either be configured or received as input. If not, the chart will look pretty boring if the time diff is in minutes and we counted everything as one block in the hour window.

I coded the count logic in java and sent back the data to UI javascript layer. Will share snippets of these both.

Java function (some parts are not implemented yet!):

PS: It’s best to implement this in JS rather than Java – Will save on i/o with just the List<String> to be sent to UI JS.

private List<List<Long>> timelineCount(List<String> timeline) {
  
  List<List<Long>> countList = new ArrayList<List<Long>>();
  
  //Sample datetime format from twitter: Wed Apr 30 14:06:49 +0000 2014
  //Convert the string datetimes to JS millis format
  List<Long> timelineJs = new ArrayList<Long>();
  final String TWITTER = "EEE MMM dd HH:mm:ss Z yyyy";
  SimpleDateFormat sf = new SimpleDateFormat(TWITTER);
  sf.setLenient(true);
  Long minTs = null;
  Long maxTs = null;

  for(String d: timeline){
    try {
      Date date = sf.parse(d);
      timelineJs.add(date.getTime());
      if((minTs == null) || (maxTs == null)){
        minTs = date.getTime();
        maxTs = date.getTime();
      }
      else{
        if (minTs > date.getTime())
          minTs = date.getTime();
        if (maxTs < date.getTime())
          maxTs = date.getTime();
      }
      } catch (ParseException e) {
        e.printStackTrace();
      }
    }

    Map<Long, Long> countMap = new HashMap<Long, Long>();
    Long newMinTs = minTs;

   //Convert this to countlist - find min, max, count stats and decide the count window
   intcount = timelineJs.size();
  //If no ts
  if(count==0){
    //TODO
  }
  //If only one ts
  if((count >= 1)&&((maxTs - minTs)==0)){
    //TODO
  }
  //If more than one ts
  if((count > 1)&&((maxTs - minTs)>0)){
    Long timeDiff = maxTs - minTs;
    Long incr = null;
    //if timediff in secs
    //5 means if less than 5 min, count is grouped every second.
    if(timeDiff < 60000 * 5){
      incr = 1000L;
    }
    //mins
    else if(timeDiff < 60000 * 60 * 5){
      incr = 1000L * 60;
      //go back to the nearest lower min
      newMinTs = minTs/(1000*60);
      newMinTs = newMinTs * 1000 * 60;
    }
    //hours
    else if(timeDiff < 60000 * 60 * 24 * 5){
      incr = 1000L * 60 * 60 ;
    }
    //days
    else { 
      incr = 1000L * 60 * 60 * 24;
    }
    for(long i = newMinTs; i<= maxTs; i = i+incr){
      //init countMap
      countMap.put(i, 0L);
    }
    //iterate timelineJs and add counts appropriately
    for(Long t: timelineJs){
      Long diff = (t - newMinTs) / incr;
      Long key = newMinTs + (diff * incr);
      Long value = countMap.get(key);
      countMap.put(key.longValue(), value + 1);
    }
  }
  //convert countMap to countList for JS in UI
  for(Long t: new TreeSet<Long>(countMap.keySet())){
    List<Long> coordinate = new ArrayList<Long>();
    coordinate.add(t);
    coordinate.add(countMap.get(t));
    countList.add(coordinate);
  }

 return countList;

}

JS code:

ajax_response contains an Object called timelineStats.

The keys of timelineStats are the plot labels. Iterating over every key in timelineStats, we get the key (label) and value which is the Array of Arrays (countList in the Java code.) We draw a plot for each label.

function draw_timeline(ajax_response){

 var labels = Object.keys(ajax_response.timelineStats);
 var df =[];
 for(var i=0; i<labels.length; i++){
 df.push({
 data: ajax_response.timelineStats[labels[i]],
 label:labels[i]
 });
 }
 var options = {
 xaxis: {
 mode: "time",
 timeformat: "%Y/%m/%d %H:%M:%S"
 }
 };

$.plot($("#timeline"),
 df,
 options
 );

}

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s