Cognos Scheduler was an extra tool you could use to schedule a variety of Cognos 7 related tasks, including DecisionStream jobs. While a handy feature, I found that virtually all of my past clients never used it. Usually there was a corporate scheduling tool or the operating system’s scheduling feature would be used instead. So this was never an issue for one of my clients until now.

With Cognos Scheduler now retired and scheduling Cognos reports simply a function of reports on the portal, there is still a service in the Cognos 10 environment that allows you to run Cognos Data Manager jobs through IBM Cognos Connection. This is the Data Movement Service.  Cognos Data Manager must be installed on your Cognos 10 server for this to work. You publish your jobs to schedule from Cognos Data Manager to the Cognos portal.

That said, most of my clients still use Windows scheduler or Unix cron jobs to schedule events like this.

Boiled down to its simplest form, DQS acts as a data “spell checker” that can apply statistical data correction, user-defined knowledge base data correction, or third party web service data correction. In the DQS interface, users define Term Based Relations (TBR) rules which can be applied against the data set. While correcting data, DQS will generate its own list of rules which you can validate.

Another relative newcomer to the SQL Server lineup is Master Data Services (MDS), which first appeared in SQL Server 2008 R2. This serves as a central repository of “golden” records, the single source of validated truth. It allows for a Master Data model to be generated and kept. This then serves as a lookup source for Integration Services packages.

SQL Server Integration Services (SSIS) is still the lynch-pin of data movement in the SQL Server environment. It integrates seamlessly with the DQS and MDS services. It has seen feature enhancements and improvements such as a catalog feature (for ease of configuration, security and management), change data capture, and built in reports for troubleshooting and logging.

Anyone interested in a deeper understanding of SQL Server 2012 should note that Microsoft will be hosting a free day-long workshop on June 22 at their Ottawa office.

But never mind all that. When it comes to contract opportunities, tools rule.

If truck drivers were hired the way ETL developers are, they would have ads like this:

Wanted: Truck Driver – must have 5 years experience driving a Ford

Most ads read X number of years in Y tool, usually at least 2 to 5 years. When you are new to a tool, even with a decade of experience in others, you can be dropped from consideration. This approach is very limiting, both to me as a contractor and to clients who are hiring. Their pool of applicants is limited, and they eliminate an excellent resource like me!

Being a self-taught computer techie in Delphi, VB.Net, Crystal Reports, Gupta databases and to a certain extent SQL Server itself (I am formally trained in Cognos and Oracle), I am capable of ramping up on new products and technologies and delivering professional-grade, quality results in a short amount of time. But I often find myself pigeon-holed into a small group of tools, and recent experience in new products is often heavily discounted if not written off completely.

ETL development itself is very conceptual. It is more than the tool, the database, or even how the tool and database work together. I recently reduced the runtime of an ETL process by 80% by using database techniques that were completely unrelated to the ETL tool in question. This technique could have worked with almost any database or ETL tool. As such ETL solutions are not as tool driven as managers or procurement might have you believe.

This is not to say that ETL tools do not have differences, sometimes even major ones. But at the end of the day, like a truck, they get your data from point A to point B. And just like any quality, experienced truck driver, ETL developers should be able to change their vehicle of choice and drive it home.

Tableau is an interesting OLAP-type analytical product that that falls in the upper-left quadrant, qualifying it as a “challenger” in Gartner-speak. But those that like it like it a lot – Gartner goes on to describe it as “The ‘sweetheart’ of the quadrant.” Apparently customers love this product.

I took a quick look at the desktop version of the software, which is offered as a fully-functional 14 day trial.

Tableau has an interesting history. It was started by folks with a strong interest in data visualization. From the beginning Tableau was positioned as a tool that would enable fast visual representations of data (original founders included a founding member of Pixar.) Tableau advertises itself as “a stunning alternative to traditional business intelligence”, attempting to carve out a niche in an area that Cognos, for example, has traditionally not been great at (in my opinion visualization has always been clumsy in tools even as advanced as Report Studio.)

Another area Tableau claims to excel in is in raw speed – “Bring your data into Tableau’s high performance data engine and work with it at blazing speed. And do it with a click—there’s no programming required. Tableau turns millions of rows of data into answers at the speed of thought.” goes the sales pitch. No “programming” required, but definitely some thinking.

When you start doing analysis with Tableau you are offered the ability to connect to a wide, impressive range of data sources. These include Excel, the usual commercial databases etc, but also open-source favourites MySQL and PostgreSQL. As well, there is an option to connect to Cloudera HADOOP Hive. Tableau is plainly positioning itself for “Big Data”-type analysis.

When you select a relational-type data source, such as Microsoft SQL Server, you have the option to select one or more tables, and establish their joins using a series of dialog boxes.  From a data-modelling perspective this interface feels a bit awkward, but it gets the job done at the desktop-level…

… and with clearly-defined keys and a simple data model this shouldn’t present data-savy users with much of a problem – more on this below.

Next we have the option of either connecting directly to a the data source, or importing the data into a native Tableau format:

This is where it gets interesting. I created a MS SQL Server database consisting of 10000 customers, 50 products, and 100 million sales rows -  a very simple model, but a large overall size for my hamster-powered laptop. I then created a MS Analysis Services cube to play with. However, working from the relational model, a user can connect to the database and importing this directly Tableau’s native format – according to Gartner a column-oriented in-memory data engine. On my admittedly underpowered laptop this took a couple of hours, but performance when querying the imported data was quite impressive – it seemed at least as fast as the Analysis Services cube. This isn’t sophisticated benchmarking, but indicates that Tableau’s engine definitely has some power. Using this feature assumes that the user is comfortable arranging the hierarchies of the data themselves, instead of having a modeller do it for them in a cube.

This approach reveals something about critical about Tableau’s market – this tool is meant for people who are comfortable with the world of databases and OLAP-style structures, and for whom creating joins, hierarchies and all the rest is a natural part of the way they think about the data – but who are also the very people interested in analyzing their data. The database, the joins, the model – all of this is a means to an end, carried out, at least to some degree, by the analysts themselves. This hints at Wayne Eckerson’s observation that real analysis is often a bottom-up process, with savvy folks in the business using the powerful tools now available to them to “end run” the IT department. This tool essentially builds-in a kind of ETL between a database and a proprietary analytical structure. This isn’t mandatory, of course, and connecting to my Analysis Services cube was quite easy and natural, but this is something to think about.

As expected, visualizations are where Tableau excels. The “Show Me” tab gives the user a number of visualization options, with hints as to what is appropriate for what kind of data.

Many of the visualizations available are quite useful – for example, below I am able to visually locate a customer who is “Tier 1”, but has very low sales. Arranging this display tool seconds:

Tableau offers the user the ability to connect simultaneously to multiple data sources. Here I have 2 data sources in the “Data” tab. Contrast this with the approach Cognos takes, where multiple data sources are put together in a package that hides this from the user. Once again, the idea is that the user knows the data (and how it relates) well enough to perform these kinds of tasks – but the user can act quickly to select the data sources they want and combine them as he or she sees fit.

Digging into all of Tableau’s features is beyond the scope of this post, but this is definitely a thought-provoking product. The BI world seems to be in a never-ending struggle between quick, user-oriented tools and the more controlled, but less agile, enterprise-grade BI suites. Tableau seems to be positioning itself as a product for the highly competent analyst in a relatively small organization – or a small part of a large organization. Gartner provides some insight here: “Tableau’s products often fill an unmet need in organizations that already have a BI standard, and are frequently deployed as a complementary capability to an existing BI platform. Tableau is still less likely to be considered an enterprise BI standard than the products of most other vendors.” Tableau is not a general-purpose reporting tool – it is an analysis tool, for analysts.

Category 1535687 “K1T 4H” Parent 1535685 Levels 1535311 OrderBy Drill 1535277
Value “K1T 4H” Lastuse 20071218 SourceValue “K1T 4H” Filtered False Suppressed False
Sign False HideValue False IsKeyOrphanage False IsTruncated False Blanks False

Category 1535689 “K1T 4H2~2363″ Parent 1535687 Levels 1535405 Lastuse 20071218
SourceValue “K1T 4H2″ Filtered False Suppressed False Sign False HideValue False
IsKeyOrphanage False IsTruncated False Blanks False

Category 1535447 “1000061″ Parent 1535689 Levels 1535409 Lastuse 20071218
SourceValue “1000061″ Filtered False Suppressed False Sign False HideValue False
IsKeyOrphanage False IsTruncated False Blanks False

You can remove problem category records in their entirety. Once your problem records are removed, Cognos Transformer should be able to open the model normally.

Warning: Do take care editing Cognos model files in this manner. Please ensure you maintain backups of your files in case your manual fix goes awry.

After digging through the logs, I discovered that the key to this problem was the Cognos Bootstrap Service (CBS).  The Cognos Bootstrap Service makes sure that the main Java process is running by monitoring the shutdown port.

The log file cbs_run.log reported:

CBSBootstrapService Process ID could not be written to file. The cogbootstrapservice -stop command will not work. Use ‘kill -9′ instead.

The log file cbs_isrunning.log reported:

CBSBootstrapService Process is not running.

So if the Cognos Bootstrap Service doesn’t run at all, then Cognos Configuration will not know if Cognos is running or not.

The file that was preventing our Cognos Bootstrap Service from starting was cogbootstrap_service.pid which resides in the c10\logs directory. Our Unix account did not have rights to write to this file, which was the cause of the problem.

Once the Cognos Bootstrap Service was able to run again, our Cognos startup/shutdown issue was resolved.

In my line of work, I spend a lot of time googling Cognos error codes or investigating Cognos issues online. My online inquiries lead me to many different sources, including many other technical Cognos blogs. What I have discovered at my last two clients is interesting – as part of their “social media” policy, they block access to many blogs hosted on most blogging websites. Essentially, almost any free blogging site is deemed “inappropiate” as per their acceptable web use policy. Of course, they do suggest that you can apply for access to the desired website if you deem it “vital” to your work. But who has time for that? By then I have skipped ahead through the search results, looking for another answer.

I can’t say for sure how widespread this practice is, but if you want maximum exposure for your online material and corporate clients are a key part of your target audience, you may want to consider hosting your blog somewhere under your own registered domain. Otherwise, you may be missing wide swaths of users that work under such a social media policy. While we could argue against such a blanket ban that equates any time spent on a blog with that of time spent on Facebook’s Farmville, I suspect it will take the corporate world some time to buy into the value of professional blogs. In the meantime, make your blog look more professional by hosting under your own domain and then you should be able to get through to everyone.

A second option, although not elegant, is to simply publish your model for each environment, changing the schema in the data source connection as you go.  Although this will work, it feels more like a work-around than a solution.

You can also choose to manage them by using Parameter Maps in Cognos Framework Manager.  If you choose to manage them in FM, the first step is to determine your environment.  This can be done by creating a query subject that will determine your database connection or database server.  A very useful Oracle function for this purpose is SYS_CONTEXT,  which can tell you the server name, the database name or a variety of other database environment identifiers.  It is also a widely available function, which means you should be able to call it no matter what Oracle account you are using.

Your Environment Query should look something like this:

You can then set up a Parameter Map to read your Environment Query.  Base your Parameter Map on your defined Environment Query, using the SCHEMA column as the key and the SCHEMA_VALUE as the value.

Once you’ve defined your parameter map, you need to make two final changes.  First, remove any Schema definition in your Data Source properties.  Do not attempt to call your Parameter Map schema value here.  If you try, you will get a recursive call error as this code will attempt to run for your Environment Query, which needs to run to get your schema value in the first place.

Next, add your parameter value to each Query Subject, except for your Environment Query:

Now you should be able to import and export your data model at will across your database environments without making schema changes at each turn.

photo credit: Patrick Haney

While researching my learning plans for this year, I came across an interesting educational opportunity in the realm of Business Intelligence. It is the Master’s of Business Intelligence program at Saint Jospeh’s University in Philadelphia. It is offered part-time and online, making it a possibility for working professionals. A quick check of my LinkedIn network showed me that of 6 registered alumni of Saint Joseph’s Haub School of Business, one has this degree – and he now works as head of Business Intelligence at the Walt Disney Company.

The program appears to have exacting but flexible admission requirements. The GMAT requirement can be waived if you hold one of a list of professional designations. It takes 20 months to complete the entire program part-time, and total tuition costs are currently $25,980 (no additional fees for foreign students, at least for the online delivery of the course). You can expect to spend between 10 and 15 hours a week on a part-time basis. A thesis is required.

From what I can gather, this program appears best suited for those intending to climb the Business Intelligence ladder all the way to the top. Think of this as a BI equivalent to an Executive MBA. It has the potential to break you into senior management. If you have the time, energy and ambition, this could be a good fit for you.

The article that alerted me to this program was a blog entry by Richard Herschel, the Chair of the Department of Decision & Systems Sciences at Saint Jospeh’s University. He has an Expert Channel on the BeyeNetwork.

Now to the question of a union operator. The first and obvious way to do this is to simply put a union directly in your SQL query. This will give you the desired result, but it is not a best practice.  For the sake of simplicity, let’s call this the wrong way.

A better way to do this, though less obvious to novice users, is to create a Query Set. Simply select the two tables you wish to union, then click on Actions and Define Query Set. You can do Union, Intersect and Except operations in a Query Set. You are limited to two tables at a time, but you will end up with a cleaner model if you take this approach. If you need to join multiple tables, you may be better off taking your desired operation back to the database in a database view or a merged table with ETL.  Any time you find yourself coding heavily complex SQL code in Cognos Framework Manager, ask yourself if an ETL solution could serve your FM data model better.

So, as usual, there is more than one way to complete a particular task in Cognos. It’s just some ways are better than others.