Exchange Reports Project Overview

During this summer, in-between semesters I was fortunate enough to get a software development job for a local company just 10 minutes walk from my door. The project was to produce an ‘Exchange Reports’ system that would provide email messaging statistics exactly to the customers specification. The system would be automated so that after reports were designed, they would be generated programmatically by a service and emailed to any recipients that had been setup to receive each report. The solution was to be comprised of 3 distinct programs that would need to be developed along with configuration tools to setup the non-GUI processes in the solution (namely the services).

I have produced the following diagram to demonstrate the solutions processes, ( Click to enlarge):

The design was in place when I started and an existing code-base was also present, but still required the vast majority of the functionality to be added. It was the first time having worked professionally as a software engineer and therefore also the first time getting to grips with existing code made by developers no longer around. More so, understanding the solutions technical proposal well enough to execute exactly what the customer and my employer wanted. I think working in IT professionally for a lot of years certainly helped me get into a comfortable stride after an initial information overload when taking on solely what was a surprisingly large but beneficial technical project compared to what I had envisioned. Being thrown into the deep end is probably the fastest way you can improve and I feel that above all, I have taken a lot from this experience which will prove valuable in the future. I’m very pleased with the outcome and successfully got all the core functionality in and finished in the time frame that was assigned. I whole heartily would encourage students thinking of getting professional experience to go for it, ideally with an established company from which you can learn a great deal. Having experienced developers around to run things by is great way to improve.

Now onto the technical details. The project was coded in C# and used WinForms for initial testing of processes and later for the configuration programs. I used a set of third-party .NET development tools from ‘DevExpress’ that proved to be fantastic and a massive boon to those wanting to create quick and great looking UI’s with reporting functionality. SQL Server provided the relational database functionality, an experience I found very positive and very much enjoyed the power of Query Language when it came to manipulating data via .NET data tables, data adapters, table joins or just simple direct commands.

Using the diagram as a reference, I’ll briefly go through each process in the solution for A) those interested in such things and B) future reference for myself while it’s still fresh in my mind because i’ll likely forget much of how the system works after a few months of 3D graphics programming and Uni coursework :P.

Exchange Message Logs: 

In Exchange 2010 Message Tracking logs can be enabled quite simply and provide a wealth of information that can be used for analysis and reporting if so desired. They come in the form of comma delimited log files that can be opened simply with a text editor. They have been around a lot of years and in the past during IT support work I have found myself looking at them from time to time to diagnose various issues. This time I’d be using them as the source of data for a whole reporting system. The customer was a large international company and to give an example from just one Exchange system they were producing 40 MB-worth of these messaging logs each day. With these being effectively just text files that’s an awful lot of email data to deal with.

Processing Service: 

The first of 3 core components of the solution, the Processing Service as the name suggests is an install-able Windows Service that resides on a server with access to the Exchange Messaging log files. The service is coded to run daily at a specified time and it’s purpose is comprised of 5 stages:

1. Connect to the Exchange server and retrieve a list of users from the Global Address List (GAL). This is done using a third-party Outlook library called ‘Redemption’ that enables this information to be extracted and then check it for any changes to existing users and/or any new users. The users are placed in a table on the SQL database server and will be used later to provide full name and department information for each email message we store.

2. Next, each Exchange Message log is individually parsed and useful messaging information is extracted and stored into various tables on the database server. Parsed log file names are kept track of in the database  to prevent reading logs more than once.

3. Any message forwards or replies are identified and tallied up.

4. A separate Summary table on the database is populated with data processed from the prior mentioned message tables. This table is what the reports will look at to generate data. Various calculations are made such as time difference between an email being received and then forwarded or replied to gauge estimates of response times being just one example; a whole plethora of fields are populated in this table, much more than could comfortably fit on a single report. Due to this large amount of potentially desirable data we later allow the user to select which fields they want from the Summary table in the ‘Report Manager’ if they wish to create a custom report or alternatively and more typically, they use predefined database ‘Views’ that have been created for them based on the customers specification which allows them to access only the data they need. Database Views are a really neat feature.

5. The databases Messaging tables are scoured for old records beyond a threshold period and deleted. This maintenance is essential to prevent table sizes growing too large. Their associated Summary data that has been generated is still kept however but I added functionality to archive this by serializing this data off and deleting it from the database if required.

Report Manager:

Initially we had thought to utilise DevExpress’s ‘Data Grid’ object controls in a custom Form application but we decided that the appearance of the reports that were generated from this were not satisfactory. This turned out to be a good design decision since we later discovered DevExpress has remarkable reporting controls that allow very powerful design and presentation features that completely overshadowed that of the Data Grids. After some migrating of code from the old ‘Report Manager’ program and having to spend a day or two researching and familiarising myself with the DevExpress API I had a great looking new application that the customer will be using to design and manage the reports.

Report Manager program

Report Manager program

The Report Manager allows you to design every aspect of a report through an intuitive drag and drop interface. Images and various graphics can also be added to beautify the design, though that wasn’t something I did nor had the time to attempt! The data objects can be arranged as desired and the ‘data source’ information for the report is saved along with it’s design layout via a neat serialization function inherent to the ‘XtraReport’ object in the DevExpress library which is then stored in a reports table on the database server for later loading or building. You can also generate the report on-the-fly and export it into various formats such as PDF or simply print it. Another neat built-in feature is the ability to issue SQL query commands using a user-friendly filter for non-developers in the report designer which is then stored along with the layout, thus the user designing the report has absolute control over the data i.e a quick filter based on Department being “Customer Services” would return only that related message data without me needing to code in some method to do this manually like was the case when using the Data Grids.

In the top left you’ll see specific icons that provide the necessary plumbing for the database server. ‘Save’, ‘Save As’ and ‘Load’ respectively writes the serialized report layout to the database, creates a new record with said layout or loads an existing saved report from the database into the designer. Loading is achieved by retrieving the list of report records stored in the reports table and placing it into a Data Grid control on a form where you can select a report to load or delete. The ‘Recipients’ button brings up the interface to manage adding users who want to receive the report by email, this retrieves the user data imported by the Processing Service and populates a control that allows you to search through and select a user or manually type a name and email address to add a custom recipient. Additionally, upon adding a recipient to the report you must select whether they wish to receive the report on a daily, weekly or monthly basis. This information is then stored in the aptly named recipient table and then relates to the reports via a reportID field.

Report Service:

Nearly there (if you’ve made it this far well done), the last piece in the solution is another Windows Service called the ‘Report Service’. This program sits and waits to run as per a schedule that can be determined by a configuration app that i’ll mention shortly. Like the Processing Service, as part of it’s logic, it needs to check if it’s the right time of the day to execute the program, of course the service continuously polls itself every few minutes to see if this is the case. Upon running it looks to see if it’s the right day for daily reports, day of week for weekly reports, or day of month for the (you guessed it) monthly reports. If it is, it then it runs and grabs the ‘joined’ data from the reports and recipient tables and proceeds to build each report and fire them out as PDF email attachments to the associated recipients. It makes a final note of the last time it ran to prevent it repeatedly running on each valid day.

Configuration Tools:

Two configuration apps were made, one for the Processing Service and one for the Report Service. These two services have no interfaces since they run silently in the background, so I provided a method via an XML settings file and the two apps to store a variety of important data such as SQL connection strings, server authentication details (encrypted) and additionally also through the need to provide certain manual debugging options that may need to be executed as well as providing an interface to set both services run times and the report delivery schedule.

Screens below (click to enlarge):

So that’s the solution start to finish, depending on time I’m told it’s possible it could be turned into a product at some point which would be great since other customers could potentially benefit from it too.

The great thing about a creative industry like programming, whether business or games, is that you’re ultimately creating a product for someone to use. It’s nice to know people somewhere will be getting use and function out of something you have made and just one reason why I’ve thoroughly enjoyed working on the project. I’ve learned a lot from my colleagues while working on it and hope to work with them again. You also get a taste for real life professional development and how it differs in various ways to academic teachings, which although are very logical and sensible are also idealistic (and rightly so) but in the real-world when time is money and you need to turn around projects to sustain the ebb and flow of business, you have to do things in a realistic fashion that might mean cutting some corners when it comes to programming or software design disciplines. I always try my best to write as clean code as possible and this was no exception but ultimately you need to the get the project done first and foremost and it’s interesting how that can alter the way software development pans out with regards perhaps to niceties like extensive documentation, ‘Use Case’ diagrams and robust unit testing potentially falling to to the wayside in favor of a more speedy short-term turn around. Certainly I imagine, larger businesses can afford to manage these extra processes to great effect, but for small teams of developers it’s not always realistic, which I can now understand.

The “dumbing down”of the games industry

Technology has moved on in the games industry, that’s for certain. Hardware, programming languages and business processes have all improved i’m sure many would agree, but does the Nth fold increase in technology also translate 1-to-1 to game play and design?

I’ve been thinking a lot about that question and I’d first like to set some context by going back to a time before PC gaming was conceived or even the first 90’s era consoles were around to change the demographic of the average games consumer forever. The days of the Commodore Amiga in fact is what I want to go back to, an era that few under the age of 25 will have ever experienced during it’s peak. The Amiga i’m confident in saying was massively ahead of it’s time in terms of hardware and gaming innovation, and not just a little bit. Built on top of the great success of it’s precursor the Commodore 64, it’s perhaps unsurprising why the system has such a mythical “stuff of dreams” status now, like did it really happen or was it just my imagination?

A Past Era:

Launched in 1985 (Amiga 1000), specs wise it featured an 8-bit 4 channel stereo sound chip, CPU co-processors (unheard of at the time) and graphics capable of up to 4096 colours at a max resolution of 640×512. These specs were incredible and it took other systems such as the NES or PC DOS gaming over 7 years to get on par with the Amiga. Now it’s all good listing specs but lets put that into perspective by comparing with another system of the day:

Shadow of the Beast – Amiga – 1989

Ninja Gaiden 2 – 1990 – NES

For reasons like the comparison above, it’s startling to me that so few gamers today have perhaps even heard of the Amiga, and strange how the NES and Sega Master System shook the world of gaming forever when they arrived despite being hugely inferior. To me as a kid in the early 90’s, I looked at the NES and thought…whats the big deal, I’ve been playing better looking and sounding games then that for years!  Shrugged my shoulders and went back to playing my dads Amiga 500. I guess looking back I was lucky to have access to an Amiga and be part of the game hobbyist scene back in the day when your average person just didn’t play computer games.

Ultimately hardware isn’t everything and the reason why the consoles made such an impact boils down to price and the fact that children could have one in their bedroom (myself included). Gaming wasn’t just for powerful multimedia systems anymore, consoles brought relatively cheap systems that every family could afford to have and thus marked the final death knell of the Amiga platform by the mid 90’s. Commodore had squandered a huge technological advantage for years and it’s failure to react to rising competition brought it to it’s knees. It’s also worth noting that as a games platform the Amiga was massively successful in the UK and across Europe, but did less successfully in the US primarily due to a larger interest in the Japanese arcade gaming culture rather then home computing. Thus the majority of Amiga games (of which there are literally thousands) were made in Europe and in fact the UK pioneered much of the games programming advances of the age that led to some greatly successful games. British studios like Sensible Software and the Bitmap Brothers, and publishers like Psygnosis are legendary and we owe them a lot for what they achieved back in the day, much of which is taken for granted now and forgotten as the fast moving games industry moves ever on like a enraged bull, never stopping to look back at lessons already learned decades ago.

Chaos Engine – Amiga – Bitmap Brothers – Subtle complexities to a simple game

The Stifling of Innovation and Creativity:

To the topic at hand and the question I started the article with. Has game play and design regressed since those days and if so why? Bluntly and unequivocally yes in my opinion,  but the why of it will take some explanation. To understand why you have to look into the past of gaming hence my above context on the Amiga, it’s unavoidable and not simply nostalgic musings. It’s the logical thing to do when analyzing something that has been great in the past, and has become less great over time. As admitted, graphically things have improved, but the root of problem is something that has caused a stifling of innovation leading to regurgitation of the same copy-cat game over and over with different artwork for years on end. The end of the 90’s was perhaps the last true great period of games innovation and creative freedom that professional games developers had. You only have to look at the quality titles released on the PC between 95 and 99 to realise this.

I’ve researched various articles and read interviews featuring leading people who worked in the earlier days and you see similarities in how they view the industry and how it has changed for developers. The core of it seems to be due to the refusal of the increasingly powerful publishers to fund games that at not a 100% safe bet (Call of Duty, Halo etc) and this has led to a massive drop in innovation that is only now perhaps being turned around by the injection of new creative blood by the Indie developer scene. Fueling the increasingly tight and controlling grip of publishers is the increasing vast sums of money that the industry now generates. Many people ARE aware of the lack of innovation but perhaps feel that there’s just no ideas left? Well there’s plenty of ideas around, the problem is that no large publisher would touch it unless it’s proven and that’s the crux of it.

Populous 2 – Amiga – Bullfrog

John Hare, a founder of Sensible Software (one of the biggest and most successful games company’s of the 80’s and early 90’s) gave a brilliant frank and interesting interview on You Tube where he discusses that during those days, publishers were happy to have talented people on board and they pretty much left you to make what you were passionate about and encourage you to push your creativity. It’s not surprising then that if you were ever motivated to go back and play Amiga games now and get over the aging visuals, you’d find a myriad of game genres, some still today undefinable such was the creative freedom back then. This issue of publishers forcing developers to copy existing games, adding just a new paint job is paramount to what is holding back the games industry in my opinion. Yes there’s Kickstarter and Steam Greenlight and they are all well and good, but I feel that the large publishers need to have a dramatic culture change if were are ever truly going to return to a golden age of innovation in game play concepts, design and execution. Perhaps the Indie scene will be the catalyst that fuels the publishers to change and allow more freedom to professional studios?

While the Amiga had it’s day, its fair to say that it was a very 2D orientated  platform and with the coming of 3D and it’s dominance in professional studios it’s not surprising that small man teams of maybe 3 or 4 can no longer produce the par standard graphical expectations in games expected for modern AAA title publishers, whom require dozens of developers and artists and millions invested to produce some of the photo realistic wizardry modern shelf titles feature. But are the incredible graphics and animation a fair trade for the disadvantages it brings?

Level design is something that has most certainly suffered from the introduction of vastly detailed environments now expected in any FPS game. It’s a simple matter of complexity, the more you introduce into a scene, the longer it takes to produce. The longer it takes the less time you have to make complicated and intelligent level design. Thus many “on rails” shooters are just that, a monorail ride with the occasional dead end to “confuse” said player and following satellite navigation way points that show up on your automap, even if the game is set in a medieval fantasy universe *cough* Skyrim.

Personally speaking photo real graphics are not a fair trade and ultimately it’s the game play that keeps you playing a game long after you’ve become desensitized to the pretty visuals. Many hugely successful Indie titles have shown this, surely it’s time for the big AAA studios and publishers to say “let’s strip down the cluttered visual complexity, take a risk and focus on game play “. Wouldn’t that be something? That and actually playing games rather then spending 30% of your time watching dialogue cut scenes. At times I think games have forgotten their roots in the arcade, and have borrowed far to heavily from Hollywood.

A change in audience & social gaming:

Another key factor in the the evolution of the games industry is tied with in turn the evolution of it’s audience. Back in the Hobbyist days of gaming, a period i’d widely class from 1980-1999, most people who sat indoors playing video games were looked at a bit strangely. They were geeks, nerds, predominantly male and it most certainly wasn’t a cool thing to do. They were probably above average at school and i’d be as bold to say statistically more intelligent or at least have an intrigue in things they didn’t understand. This would manifest itself in a way that if you presented a challenging game to a geek, they would be much more likely to try and figure it out and spend time trying to overcome the complexities, like a piece of homework or a maths question. A less motivated individual with less intrigue would put the game down, upset about it being too hard and never play it again. Therefore the audience in a nutshell back then was more mature and forgiving about games and it allowed a degree of freedom to developers to really go to town on sophisticated game play elements that would take time to master and learn, but ultimately paid off long term over simple repetitive games.

Now as pretty much most are aware, nowadays games on the whole are streamlined and simplified for the new average audience demographic, whom is not a geek, nerd or in fact *shock* actually male. Social gaming has brought women into the gaming consumer audience and rightly so, women should be part of it. Men too have lapped up the new social gaming phenomenon but irrespective of gender which is irrelevant, the key point is that the “nerd gamer” is no longer the average demographic and thus games are now being effectively aimed at less patient, casual orientated “non-gamers”. Social games are not games in their truest purest sense, they are not escapism, or adrenaline pumping or a visual feast or inspiring, they are simply a feedback-response stimulus loop that passes time for the bored individual. Engineered game play featuring staggeringly simple repetitive tasks with a carrot style reward at the end. Real games ARE more then that aren’t they? I think so.

Conclusion:

The whole evolution of the industry is a double-edged sword. It’s not all bad certainly, there’s never been an easier time to get into the games industry and there’s certainly a lot more jobs around with better pay then there used to be, however along with vast sums of money has come the bureaucracy that is rife within what is essentially a creative industry and there are startling parallels with the movie industry. Like with games, the increasingly powerful few have begun to control too much of what directors make and the many unneeded remake movies are effectively synonymous with the copy-cat games made today in the games industry. But, I wont lay the blame just on publishers. John Hare mentioned something regarding the fact that the industry is saturated with content and most of it not good or to a high enough quality. This waters down the expectation of what a good game actually is, and with more and more game developers coming into the mix this could spiral further. His solution? Less developers/designers and who are to a higher standard. Is that the answer? I’m not sure but poor games will in-turn inspire more poor games, it’s a vicious circle that we must break and ultimately in my opinion it should start from the top AAA studios and work it’s way down, not the bottom up.

It’s a topic I feel passionate about and there’s no easy answers but that’s my take on it and an opinion from someone who has played far too many games over the past 29 years and hope to influence the games industry in some way (even if just a nano) by making games myself. I hope that in time, developer creativity will flow however it wants wherever it wants and only our imagination will limit where games can take us.

Lemmings – Amiga – DMA Design