How old is your data warehouse? It’s a simple question and probably one you don’t think about much. The majority of production data warehouses are now 15-20 years old and probably very transactional centric. Over the years, you’ve probably remodeled “the house” more than a few times—adding some “rooms” and “upgrades” here and there. It’s starting to feel its age as more Business Intelligence requirements have been added, including Mobile applications and specialized analytics. And more and more ideas seem to show up in your inbox every day, especially Big Data questions.
iOLAP's Insights
Have you ever wondered how it would feel to stand on the top step of an Olympic podium, lean over and have a gold medal placed around your neck? You not only have family, friends and coaches cheering you on, you have a whole nation behind you. It must be an overwhelming moment. While few ever have that opportunity, you can be the go-to champion in your organization. How can that happen?
With a user experience background, I take usability seriously when designing and developing reports. Usability provides end users with polished reporting that is intuitive while still informative. Paying attention to these details can make a huge difference.
For example, “stop light” thresholds are a great conditional formatting style to quickly identify key indicators. They can help users identify which region is growing fastest or is poorly performing.
A report uses red, yellow, and green circular indicators that are commonly associated with the flow of intersection traffic (red for warning or stop, yellow for caution or slow and green for go).
While these stop light colors are known everywhere, what if the end user is color blind?
Within SSIS a FTP task exists which enables you to access a FTP server. However, it does not support Secure FTP (SFTP). There are many Secure FTP tools to choose from and each requires a unique set of commands and designs to be used within SSIS.
For this article, I am using Putty.org’s file transfer tool commonly known as PSFTP (a free open source tool). This tool performed as needed and we were able to build additional processes around it to enhance its use and satisfy our customers’ requirements.
Click here to see the January 2014 iOLAP Insights newsletter.
Like most of you, I work in the corporate world. I’ve been around long enough to be part of good teams and bad teams. I have also had the opportunity to build teams. Building a team is challenging and a lot of hard work. Being on a bad team is a stressful nightmare. Building a bad team is, well, a long story. If you did it once, you’re probably no longer with that company. With most companies, you are either a player or you are a coach (boss). If you’re self-employed that can be the most challenging—because you’re both.
I am also a big football fan. High school, college, professional, fantasy—I like all of it. I never played on the field myself but I love watching a great game at any level. As the season winds down at this time of year, I always get a little sad that it will be eight months before I get to watch my favorite teams again.
Starting in the late 1970’s and continuing throughout the decade of the 1980’s, one of my favorite football coaches was Bill Walsh.
So what exactly is Big Data?
In the real world view, Big Data is the culmination of several years’ worth of data that your company has stored in their data warehouse as instructed by their DBA since, well, forever. This data that has been archived in different locations for safe keeping, and possible later use, is extremely valuable for marketing, sales and other decision makers in your organization.
The official Wiki definition of Big Data is: “a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization.” You will see that definition used in a lot of places.
Maintaining a database of postal mail addresses requires them to be maintained over time and validated. Conventional wisdom is usually that an address database should be largely static and never change after data entry unless someone physically moves. However, as in many cases, conventional wisdom is not correct in this case. Both the United States Geological Survey (USGS) and the United States Postal Service (USPS) recommend that physical street addresses be maintained at least once a quarter via a process named geocoding, which includes street name, city, state and zip code validation.
Click here to see the December 2013 Issue of iOLAP Insights.
Amazon offers loading data to Redshift either from flat files that are stored in an Amazon S3 bucket or from Amazon DynamoDB table. In this article, I will show you how to load data from your local machine to Amazon Redshift using the Amazon S3 service. Also, for inserting data to Amazon Redshift, I will show you how to use COPY command.