
Kevin Connolly	Geek. Developer. Entrepreneur. All-around good guy. Builder of things. Solver of problems. Changer of worlds.

Three Weeks, Source Control, Project Rename

Posted 8/6/2008 12:36:04 AM

Today marks three weeks at my new job at the Major Insurance Provider; and today I finally got access to Source Control. Thereto, I've been working.

The project I've been working on, henceforth known as Project Rename, is reformatting outgoing communications to meet a new corporate standard. Somehow, this reformatting effort is causing most of the company's outgoing electronic communications to be redirected through Project Rename.

Now that I've gotten my feet wet in the project, let me provide you with an overview of the goal:

Receive data feed from several potentially-unknown sources. These files describe which E-mails to send to which addresses.
Based on the mapping described between E-mail addresses and messages to send, perform specific lookups and retrieve the results.
Organize the acquired data and send out E-mails once they've been compiled.

And the process they use to perform these basic goals:

Input data is created, usually by an external group (You may know them as "Upstream Dependencies"), but sometimes by us via an SSIS job.
Receive flat files via FTP, in one of four formats
- Fixed-width
- Pipe-delimited
- XML
- A hybrid format which is fixed-width until about 50 characters before the end of line, then has a dynamic number of pipe-delimited fields, but only some of the time. Luckily, we don't use any of the fields that far over.
Run an SSIS job which parses the input file and loads it into SQL
A Windows version of the Mainframe CA-7 Scheduler runs on a Windows server somewhere. This scheduler performs some sort of trigger operation which causes the next step to begin.
A console application known only as The Green Screen executes. It retrieves the list of messages from SQL (which came from the aforementioned upstream dependency), runs an XSL transform against a predefined XSLT template (Previously entered manually by hand; but at the low rate of 1-2 templates per week). The console app then loads the result back into SQL.
A Windows Service, running on 12 Application Servers, retrieves the jobs and another XML feed that contains instructions on what to do with these messages. These instructions include:
- Filtering: A complex series of rules which specify whether or not to "filter out" this message, thus removing it from the mail queue.
- Lookups: Generally performed through a WebMethod, but sometimes performed through pure code and database access. Instructions on information that must be looked up, such as:
  - Have we sent this E-mail to this person this month? If so, we don't want to send it again.
  - Has this person opted into or out of this E-mail option? Is this E-mail option's default on or off?
  Obviously, the Lookups functionality is tied directly into the Filtering functionality.
- Assorted Aggregators, which perform miscellaneous functionality as defined by Class Name embedded directly within the XML instructions.
Finally, the messages are sent to the SMTP server via a WebService.

My inner Architect is crying. I'm really learning the difference between Developers and Programmers. At my last job at A Major Telecom, I was a Developer. Development includes architecting, optimizing, and always improving your existing ideas while continuing to come up with new ones. As a Programmer, I am encouranged to follow their existing architecture, which is only half-built as is. I could go off on a rant about how Agile is supposed to "Embrace change", but I digress.

My Books:

(Amazon Author Page)
$A Comprehensive Review of the Federal Budget:\nOur Forefathers' Solution to Tyranny$
[ Kindle | Paperback ]

Liberty: How America's increasingly socialist agenda impacts us all

[ Kindle | Paperback ]

Browse:

Social: