metafy downloads: our own dogfood


In this example, Metafy is going to eat its own dogfood by using Anthracite to help us process the e-mails that we receive when you sign up to download the trial software from us.

We receive the e-mail in Mail.app, then we periodically save the thread to a text file (say, every 100 messages per file). There's a seperate file for all download requests for each different software product, and the most recent version of those files are stored in a "Sources" directory (we archive them elsewhere when done).

This example is easy to understand and simple to construct on your own, plus it shows the power of a few simple actions in Anthracite, and solves a crucially important problem in our business: how to extract information from e-mails.

Here's the completed process chain, click for a full-size image:




In this process, the Source Object (red round sphere) is loading the files that are in the "Sources" directory.

We then pass the the text from those sources directly into four different processors. Three of these are pairs of processors to extract a particular line that starts with a term, and then remove that term, leaving behind only what is unique about the line.

Here we see a pair of these processors for the "From" line in the e-mail:


The First of the two in the pair is a Regular Expression Processor Object:



This is a pretty common task, but Regular Expressions may be a little tricky for some folks. Here's a closeup of the Regular Expression used to collect this particular bit of info from the incoming text:



Note that the Regex is comprised of the word "From" followed by a Colon and a Space, then the Regex special chars ".+" signifying a indeterminate string of characters up to the "$" end of line.

Then, once we've found that line, we don't need the "From: " anymore, so we remove it with the second of the pair of processors, a "Find Replace" processor that looks for the "From: " and replaces it with nothing:



We repeat that same basic pair of step for "From", "To" and "Date", but "IP" is a more easily identified pattern in the text, so we can use a Regular Expression to collect that information without any other identifying text around it, so it doesn't need the Find/Replace cleaning treatment.

Now that we've configured how to process these four useful pieces of data from our incoming e-mails (and remember that if we get 100 messages coming in, each of these processors will churn through them and produce 100 pieces of output), we're ready to do something with this information.

To demonstrate the flexibility and ease of using Anthracite, we're going to output the information we've gleaned from the e-mails into three different formats, and do one of those formats in two different ways.

At the bottom of the overview, you can see two Export Objects (green cones) and two Results Objects (blue boxes), these are the four different outputs we're going to create.

From the left: MySQL Export, CSV File Export, CSV Report Storage and HTML Report Storage.

It's IMPORTANT to note that the Export and Report Objects both use the names of the inputs to know how to distribute and use the pieces of data that we've been collecting. So, you'll want to be sure to pay close attention to your object naming, for example, your object names will have to be valid field names for your database.

First, the MySQL Export - we collect this info into a MySQL database to integrate with our other customer processing systems. Configuration of the MySQL export is covered in other examples, but it should be noted that this presumes there are a MySQL database and client running, and that we have a user account to access them, and that it's configured with a database and table to hold these four different bits of data. You'll want to use an admin tool like PHPMyAdmin in conjunction with your MySQL database unless you prefer a lot of typing in the Terminal.

Second, the "BatchFileOutputCSV" Export Object. This object will write out a spreadsheet-compatible CSV (comma separated values) file that contains the information we've collected. Here's how we configure it:



There are a lot of fields that make it look complex, but the ONLY settings we've had to change are the Name of the Object, the Output Directory, that we want CSV output, and that we want all the information Combined in the output.

That last setting is also important to note, otherwise you will end up with separate output files for each of your results (which you may want some other time).



That's all it takes to do both MySQL output and CSV output of data that we've collected from our incoming e-mails.

But, since we're paranoid and vain, we're not going to stop there, and we're going to produce two additional reports, one as a backup of the CSV data, and another as an HTML page so we can view all of the information from these e-mails at a glance.

For both of these options, we're going to use the "Custom Snippet" Report Object, wrap the output of that report with either the CSV headers, or an HTML header for the data, and then store that in a Results Object within Anthracite.



The Custom Snippet Report Object is explained in another example, but in short, it allows us to define a snippet of text that will be repeated for each of our data elements, using a specially formatted version of the names of the inputs to the report object to trigger the replacement.

For example, the CSV Snippet for our four inputs is:


Where we simply put two "_" (underscore) characters on either side of the input name to replace the data into the report result.

Here's the HTML Snippet for the same purpose, this one generating table rows (>TR<).




Once we've made these custom report parts, for the HTML version we wrap the snippet output inside of HTML table tags with a line of header information.



For the CSV version of the report, we use the "wrap" processor to pre-pend the field names in front of the data results.



And we use this process and others like it daily to help us run the business.

Buy Anthracite Today! and you, too, can save time and boost productivity by solving problems just like this one in your business.

 

[top] [anthracite home]

last update: 05/21/2004

Copyright © 2004 All Rights Reserved, Metafy LLC