Making Data Pretty

Beauty is in the Data, not the Tools

for the International Journalism Festival 2012

Dan Nguyen twitter: @dancow / @propublica
April 27, 2012
Shortlink: http://bit.ly/beautydata

The Good News

There's more tools and visualization options than before. Here's a totally non-comprehensive list:

Self-hosted	Third-party hosting
Highcharts D3.js gRaphael.js Protovis.js Flot.js	Tableau Google Fusion Tables Google Static Charts Google Interactive Charts Many Eyes

The Bad News

There's more tools and visualization options than before.

i.e. there's more minutiae and technicalese to remember.

Even the best tools break (browser compatibility), go out of fashion (i.e. every interactive graphic made in Flash), or end up costing too much.

Tools exist to make your life easier

The best tools share the best patterns and intelligent design. If your visualization tool has a lot of arbitrary fickleness to it, there's probably a better tool out there.

Screenshot of Google's Chart Gallery

Invest your energy into design and data intelligence, not in memorizing the tools.

What will always be the same is:

The understanding data and its quality, quantity, and depth.
The skills to collect, clean and organize data: e.g. web scraping, database queries, spreadsheet skills
Good design principles and aesthetics

The Visualization Programs are the Easy Part

Let's go over a few visualization options, show how they share the same common steps, with small difference in details.

Excel: Easy and offline

Open a data file, highlight the data, and click one of the Chart Wizards (sample data: March 2012 web browser usage share).

...and presto!

(pie charts – especially 3D ones – aren't considered the best visualization option, but you get the point)

Why not use Excel, all the time?

Pros:

Easy, point-and-click interface for handling data and charts
Lots of visualization options
Great for one-off, hand-tailored graphics

Cons:

Have to own Excel to create visualizations
Have to own Excel to edit them
No well-defined process to update graphics and re-upload them
Too cumbersome for day-to-day online production (when you have to dynamically generate dozens to thousands of charts)

Leaving Excel for other, more flexible options

Leaving the friendly point-and-click interface of Excel (or OpenOffice's or Google Docs' version of the spreadsheet) can be intimidating.

However, once you have the data in a proper format, you can port it to other visualization packages fairly easily.

What is Data?

Data is not determined by the program used to open and view it, whether it be Excel, Access, Fusion Tables, SQL Server, SAS, etc etc.

This Excel spreadsheet:

...is nothing more than this text file

It is all about separation (a.k.a. delimiters)

Data in a spreadsheet is nothing more than text separated by a common symbol (a.k.a delimiter), such as a comma, pipe |, or tab character.

Sometimes, switching from one tool to the other is as simple as doing a find-and-replace.

Excel interprets the tab characters in the text file as the column boundaries.

Save that file in Excel as .csv (comma-separated values), and the column boundaries are now commas.

Google Static Charts API

One of the simplest online visualization options. Send the service a properly formatted URL; get back a chart as image file

All of the instructions are in Google's official documentation.

How to make a Google Pie Chart

Read the pie-chart docs here

Start off with the URL for Google's service:
https://chart.googleapis.com/chart?

Specify the chart type:
&cht=p
Specify the dimensions:
&chs=300x200
Add the data:
&chd=t:10,20,30,50

The resulting URL:

https://chart.googleapis.com/chart?cht=p&chs=300x200&chd=t:10,20,30,50

Creates this image:

The important part: formatting the data

https://chart.googleapis.com/chart?cht=p&chs=300x200&chd=t:10,20,30,50

The only thing that really matters is the actual data (e.g. 10,20,30,50) and the parameter it comes after (&chd=t:)

All the other parameters are just things to copy and paste.

Redoing the Excel Chart

Using our previous dataset (March 2012 web browser usage share), we simply:

Replace all tab characters with commas
Insert into the Google Chart API format:

https://chart.googleapis.com/chart?cht=p&chs=300x200&chd=t:34.81,30.87,24.98,6.72,1.78,8.99

The result graphic:

We can add labels, as per the Google pie chart documentation:

Google Static Charts: Pros and Cons

Pros

Because the charts are images, they look the same in every browser
Flexible, relatively simple API
Easy to include in any web application

Cons

Google announced the static charts API is deprecated: support may end in 2015
All good things come to an end

gRaphael.js

A JavaScript library for creating interactive graphics inside the web browser. It requires users to have JavaScript – which virtually every user does these days.

gRaphael.js simple code

[official example]

	
<script>
$(document).ready(function(){	
var r = Raphael("pie-holder-simple");

var pie = r.piechart( 400, 300, 100, 
	[34.81, 30.87, 24.98, 6.72, 1.78, 8.99]); // the data
});
</script>

<div id="pie-holder-simple" style="width: 500px; height: 400px;"></div>

gRaphael.js: complex pie

[link to code page]

gRaphael.js: the Data Part

	// (jQuery not required, per se, just used here for consistent sequence loading)
<script src='assets/lib/jquery-1.7.2.min.js'></script>
<script src='assets/lib/raphael-min.js'></script>
<script src='assets/lib/graphael/g.raphael-min.js'></script>
<script src='assets/lib/graphael/g.pie-min.js'></script>

<script>
$(document).ready(function(){
var r = Raphael("pie-holder");
var pie = r.piechart(
	350, 200, 150, 
	[ 34.81, 30.87, 24.98, 6.72, 1.78, 8.99 ], 
	{ 
	legend: ["IE", "Chrome", "Firefox", "Safari", "Opera", "Mobile"], 
		legendpos: "east"
	}
);

pie.hover(function () {
 	this.sector.stop();
       this.sector.scale(1.1, 1.1, this.cx, this.cy);

                if (this.label) {
                    this.label[0].stop();
                    this.label[0].attr({ r: 7.5 });
                    this.label[1].attr({ "font-weight": 800 });
                }
            }, function () {
  	this.sector.animate({ transform: 's1 1 ' + this.cx + ' ' + this.cy }, 500, "bounce");

            if (this.label) {
                this.label[0].animate({ r: 5 }, 500, "bounce");
                this.label[1].attr({ "font-weight": 400 });
            }
        });

})

</script>

<div id="pie-holder" style="width: 600px; height: 500px;"></div>

gRaphael.js: the Translation Part

After you see where your data parts go, including the headers (the legend), it's a matter of find-and-replace.

Sidenote: one of the best non-programming, data-related skills you can learn are regular expressions. All it takes is printing out a cheat sheet and putting it next to your monitor.

Given:	IE Chrome Firefox Safari Opera Mobile
Find:	(.+?)\t
Replace:	"$1",
Result:	"IE","Chrome","Firefox","Safari","Opera",Mobile

[interactive regex link]

gRaphael.js: Pros and Cons

Pros:

Doesn't involve creating image files.
Doesn't rely on third-party image creator (i.e. Google, Tableau)
Very good compatibility

Cons:

May not always work on every browser ever made.
Requires loading JavaScript libraries on your webpage
Requires some knowledge of JavaScript and HTML

[download link]

Dense Data is Beautiful Data

Beauty is data deep

A visualization is only as good as the quality and depth of its data.

Great visualizations naturally come from good data, just as great stories come from an abundance of reporting and interviews.

With good data, you don't need to resort to "chartjunk" – visualization elements that decorate over shallow data (just as filler quotes are used to pad a weak story)

One variable

[dataset, Google Chart docs]

Life expectancy for each country, taken from the World Bank's 2010 dataset. Country names have been removed for space considerations

https://chart.googleapis.com/chart?cht=bvs&chbh=a&chs=700x325&chd=t:48.3,76.9,72.9,50.7,75.6,73.8,81.7,80.4,70.5,75.2,68.6,76.6,70.4,79.9,75.8,55.6,79.3,66.9,66.3,75.4,53.1,73.1,73.5,54.9,49.9,62.5,51.1,80.8,73.8,47.6,49.2,78.9,73.3,73.4,60.6,48.1,57.0,79.2,54.7,76.5,79.4,77.4,79.1,73.2,75.5,73.0,71.7,50.8,61.0,75.4,58.7,69.2,79.9,81.4,62.3,58.2,73.3,80.0,63.8,80.4,75.7,70.8,53.6,47.7,69.5,61.8,72.8,82.9,74.2,81.5,65.1,68.9,68.5,80.3,81.5,81.7,72.8,82.9,73.3,68.3,56.5,80.8,69.9,69.4,67.1,73.5,72.4,47.4,56.1,73.3,80.1,80.8,74.6,66.5,53.5,74.0,76.6,51.0,80.9,58.2,73.0,76.7,68.8,68.9,68.2,74.3,71.9,49.7,62.1,68.4,80.7,73.7,54.3,51.4,81.0,65.2,76.0,62.4,72.3,73.8,68.5,76.2,79.0,78.9,73.5,68.8,55.1,72.3,64.3,73.9,59.0,73.9,73.0,47.4,81.6,75.1,79.4,67.5,52.1,81.6,74.7,74.4,72.1,61.1,48.3,81.5,82.2,75.7,67.3,57.4,73.9,62.0,56.6,72.2,69.8,74.6,73.7,64.9,53.6,70.3,76.6,80.4,78.2,76.2,68.0,70.8,74.1,74.8,65.0,48.5,49.9

Even if country names were included, this is trivial information. Interesting at a glance, but so what?

Two variables

[dataset, Google Chart docs]

Life expectancy (x-axis) and fertility rate (births per woman, y-axis)

https://chart.googleapis.com/chart?cht=s&chds=a&chs=600x325&chd=t:48.3,76.9,72.9,50.7,75.6,73.8,81.7,80.4,70.5,75.2,68.6,76.6,70.4,79.9,75.8,55.6,79.3,66.9,66.3,75.4,53.1,73.1,73.5,54.9,49.9,62.5,51.1,80.8,73.8,47.6,49.2,78.9,73.3,73.4,60.6,48.1,57.0,79.2,54.7,76.5,79.4,77.4,79.1,73.2,75.5,73.0,71.7,50.8,61.0,75.4,58.7,69.2,79.9,81.4,62.3,58.2,73.3,80.0,63.8,80.4,75.7,70.8,53.6,47.7,69.5,61.8,72.8,82.9,74.2,81.5,65.1,68.9,68.5,80.3,81.5,81.7,72.8,82.9,73.3,68.3,56.5,80.8,69.9,69.4,67.1,73.5,72.4,47.4,56.1,73.3,80.1,80.8,74.6,66.5,53.5,74.0,76.6,51.0,80.9,58.2,73.0,76.7,68.8,68.9,68.2,74.3,71.9,49.7,62.1,68.4,80.7,73.7,54.3,51.4,81.0,65.2,76.0,62.4,72.3,73.8,68.5,76.2,79.0,78.9,73.5,68.8,55.1,72.3,64.3,73.9,59.0,73.9,73.0,47.4,81.6,75.1,79.4,67.5,52.1,81.6,74.7,74.4,72.1,61.1,48.3,81.5,82.2,75.7,67.3,57.4,73.9,62.0,56.6,72.2,69.8,74.6,73.7,64.9,53.6,70.3,76.6,80.4,78.2,76.2,68.0,70.8,74.1,74.8,65.0,48.5,49.9|6.3,1.5,2.3,5.4,2.2,1.7,1.9,1.4,2.3,1.9,2.2,1.6,1.4,1.8,2.8,5.3,1.8,2.4,3.3,1.1,2.8,1.8,1.5,5.9,4.3,2.6,4.5,1.7,2.4,4.6,6.0,1.9,1.6,2.1,4.9,5.8,4.5,1.8,4.4,1.5,1.5,1.5,1.9,2.6,2.5,2.7,2.3,5.2,4.5,1.6,4.2,2.7,1.9,2.0,3.3,4.9,1.6,1.4,4.2,1.4,2.2,4.0,5.2,5.1,2.3,3.3,3.1,1.1,1.3,2.2,2.6,2.1,4.7,2.1,3.0,1.4,2.3,1.4,3.8,2.6,4.7,1.2,2.3,2.9,2.7,1.2,1.8,3.2,5.2,1.6,1.6,1.1,1.4,4.7,6.0,2.6,1.8,6.3,1.4,4.5,1.5,2.3,3.5,1.5,2.5,1.7,2.3,4.9,3.2,2.7,1.8,2.6,7.1,5.5,2.0,3.4,2.5,4.0,3.0,2.5,3.1,1.4,1.3,1.8,1.4,1.5,5.4,3.9,3.7,2.8,4.8,1.4,2.5,5.0,1.2,1.4,1.6,4.2,2.5,1.4,2.3,2.0,2.1,4.4,3.4,2.0,1.5,2.9,3.3,5.5,1.6,5.6,4.1,3.9,1.6,2.0,2.1,2.4,6.1,1.4,1.7,1.9,2.1,2.0,2.5,3.9,2.5,1.8,5.2,6.3,3.3

Shows an interesting correlation: "Living longer with fewer children," as Google's Public Data Explorer puts it.

Three variables

[dataset, Google Chart docs]

Life expectancy, fertility rate, gross domestic product (dot-size)

Google has a URL-size limit, so not all countries are included here.

https://chart.googleapis.com/chart?cht=s&chxt=x,y&chds=a&chxs&chs=600x425&chd=t:49.9,48.1,56.1,47.4,53.5,58.7,54.3,61,49.7,66.5,53.6,47.6,58.2,48.3,53.6,56.6,57.4,68.4,55.1,54.9,47.7,49.9,51,62,61.8,68.6,49.2,60.6,55.6,62.5,56.5,67.3,69.4,47.4,65.2,59,58.2,73.7,51.1,54.7,67.1,64.3,74.8,48.5,67.5,51.4,65,63.8,68,62.4,65.1,61.1,68.9,66.3,72.8,66.9,68.5,68.2,74.7,68.5,73.3,68.8,73,71.9,72.3,70.8,75.7,70.8,68.9,69.5,57,70.3,73.8,69.9,72.3,73.8,72.2,71.7,48.3,76.9,69.2,64.9,75.5,75.8,74.6,75.4,73.3,50.7,74.6,73.3,72.9,73.9,73.2,73.9,72.8,62.1,73.8,70.5,70.4,76.6,73.4,73.5,72.1,74.3,74.4,52.1,53.1,75.7,73.5,76,73,79.2,74,62.3,75.6,68.3,76.7,72.4,73.7,68.8,73.1,73.5,73,73.3,76.2,76.2,78.9,74.2|4.3,5.8,5.2,5,6,4.2,7.1,4.5,4.9,4.7,5.2,4.6,4.9,6.3,6.1,4.1,5.5,2.7,5.4,5.9,5.1,3.3,6.3,5.6,3.3,2.2,6,4.9,5.3,2.6,4.7,3.3,2.9,3.2,3.4,4.8,4.5,2.6,4.5,4.4,2.7,3.7,1.8,6.3,4.2,5.5,5.2,4.2,2.5,4,2.6,4.4,1.5,3.3,3.1,2.4,3.1,2.5,2.3,4.7,1.6,3.5,2.7,2.3,3,4,2.9,3.9,2.1,2.3,4.5,1.4,1.7,2.3,3.9,2.4,3.9,2.3,3.4,1.5,2.7,2.4,2.5,2.8,2,1.1,1.6,5.4,1.4,3.8,2.3,1.6,2.6,1.4,2.3,3.2,2.5,2.3,1.4,1.8,2.1,1.5,2.1,1.7,2,2.5,2.8,2.2,1.4,2.5,1.5,1.8,2.6,3.3,2.2,2.6,2.3,1.8,2.1,1.5,1.8,1.2,2.5,1.6,2,1.4,1.9,1.3|192,199,247,325,339,358,358,403,410,421,452,457,466,501,509,523,524,525,530,536,580,595,602,624,671,675,676,736,749,795,795,820,847,1004,1019,1034,1044,1132,1147,1154,1177,1193,1224,1253,1261,1278,1300,1325,1381,1382,1410,1425,1631,1979,2026,2088,2140,2250,2375,2565,2621,2680,2698,2796,2840,2862,2893,2911,2946,2948,2970,3007,3031,3059,3117,3323,3349,3426,3502,3677,3708,3967,4008,4061,4199,4409,4428,4451,4461,4560,4567,4608,5215,5270,5275,5331,5401,5718,5765,6039,6240,6333,6466,6505,6884,7280,7403,7435,7539,7589,7591,7691,8373,8729,9124,9132,9133,9228,10094,10440,10710,10723,10766,11045,11633,12294,12431,12863

Another interesting, if obvious correlation. We've hit the upper-limit on what Google Static Charts can legibly display.

Four variables - with interactivity!

[dataset, Google Data Explorer docs]

X: Life expectancy
Y: Fertility Rate
Dot-size: Gross domestic product
Slider: Year (of measurement)

Now you need to do something interactive – Google has its Data Explorer tool

(Comes close to information overload, but new interesting trends are revealed)

The SOPA Opera project

Concept: Show which U.S. congressmembers have taken a stand on the controversial SOPA bill.

This project, projects.propublica.org/sopa, has the record for one-day web traffic at ProPublica.

Easy to find data:

Congressional bio: names, political party, photos (Sunlight Foundation, THOMAS, NYT Congress API)
Legislation sponsorship data (THOMAS - Library of Congress)
Campaign contributions (OpenSecrets.org)
Politicians' statements of support/opposition (personal research)

The starting point

We can do better than a text listing:

Visual feature #1

Small multiples – the repetition of simple, data graphics for easy comparison. Coined by Edward Tufte.

Visual feature #2

Simple, but snazzy interactivity – isotope, a jQuery library used for dynamic sorting and filtering. By David DeSandro.

From Text to Graphic

This next section contains all the steps needed to go from a simple spreadsheet to an interactive graphic.

It uses Google Charts and isotope.js...so some knowledge of HTML and JavaScript is needed to go beyond cut-and-paste.

Fun with Regular Expressions

But the main point of this exercise is to show how find-and-replace can move your data from one visualization tool to the other.

Regular expressions are like find-and-replace on steroids: they allow you to match patterns, not just exact words.

Regexes are one of the best all-purpose, never out-of-style concepts any data journalist can learn, without any need of programming skills. Just print out a cheat sheet.

SOPA Datapoint #1: Name

Just a simple list of legislator names

Justin Amash
Michele Bachmann
John Campbell
Maria Cantwell
Joe Baca
Michael Bennet
Jeff Bingaman
Roy Blunt
Mary Bono Mack
Saxby Chambliss
Christopher Coons
Jim Cooper
Bob Corker
Dianne Feinstein
Al Franken
Kirsten Gillibrand
Robert Goodlatte

SOPA Datapoint #1: The transformation steps

Using just the first name and last name columns of the data

[data] / [interactive link]

Given:	Justin Amash Al Franken
Find:	(.+?)\t(.+)
Replace:	<li>$1 $2 </li>
Result:	<li>Justin Amash</li> <li>Al Franken</li>

SOPA Datapoint #2: photo file

(Eyes are drawn to faces)

Justin Amash
Michele Bachmann
John Campbell
Maria Cantwell
Joe Baca
Michael Bennet
Jeff Bingaman
Roy Blunt
Mary Bono Mack
Saxby Chambliss
Christopher Coons
Jim Cooper
Bob Corker
Dianne Feinstein
Al Franken
Kirsten Gillibrand
Robert Goodlatte

SOPA Datapoint #2: The transformation steps

Using just the first name, last name, and image url columns of the data

[data] / [interactive link]

Given:	Justin Amash beauty-assets/members/50/A000367.jpg Al Franken beauty-assets/members/50/F000457.jpg
Find:	(.+?)\t(.+?)\t(.+)
Replace:	<li><img src="$3"> $1 $2 </li>
Result:	<li><img src="beauty-assets/members/50/A000367.jpg"> Justin Amash </li> <li><img src="beauty-assets/members/50/F000457.jpg"> Al Franken </li>

SOPA Datapoint #3: Yes/No on SOPA

This is derived from official sponsorship list, voting history on related bills, and public statements. Same regex as before, just moving the HTML around.

Joe Baca
Michael Bennet
Jeff Bingaman
Roy Blunt
Mary Bono Mack
Saxby Chambliss
Christopher Coons
Jim Cooper
Bob Corker
Dianne Feinstein
Al Franken
Kirsten Gillibrand
Robert Goodlatte

Justin Amash
Michele Bachmann
John Campbell
Maria Cantwell

SOPA Datapoint #4: Party

Many anti-SOPA activists assumed that SOPA was a Republican initiative. A simple color-key visualization of party membership showed otherwise.

Howard Berman
Robert Goodlatte
Roy Blunt
Kelly Ayotte
Richard Blumenthal
Lamar Smith
Robert Menendez
Dianne Feinstein
Ben Quayle
John Carter
Mary Bono Mack
Al Franken
Steven Chabot
David Vitter
Lee Terry

Ron Wyden
Anna Eshoo
Zoe Lofgren
Mike Lee
Jerry Moran

SOPA Datapoint #4: The transformation steps

This requires the use of CSS to set the colors. So, something like:

<style>		
	li.R{
		background: #FFB2AC;
	}	

	li.D{
		background: #C3D7FA;
	}		
</style>

Regular expression [data] / [interactive link]

Given:	Justin Amash beauty-assets/members/50/A000367.jpg N R Al Franken beauty-assets/members/50/F000457.jpg Y D
Find:	(.+?)\t(.+?)\t(.+?)\t(.+?)\t(.+)
Replace:	<li class="$4 $5"><img src="$3"> $1 $2 </li>
Result:	<li class="N R"><img src="beauty-assets/members/50/A000367.jpg"> Justin Amash </li> <li class="Y D"><img src="beauty-assets/members/50/F000457.jpg"> Al Franken </li>

SOPA Datapoint #5: Money

OpenSecrets/Center for Responsive Politics classifies campaign contribution data by industry/job of the contributor. Adding this datapoint lets us infer the nature of each candidate's support base.

 
Elton Gallegly
 
Kirsten Gillibrand
 
Robert Goodlatte

 
Michele Bachmann
 
John Campbell
 
Maria Cantwell

SOPA Datapoint #5: The transformation steps

[data] / [interactive link]

Given:	Barbara Boxer beauty-assets/members/50/B000711.jpg Y D 571600 348691 Charles Schumer beauty-assets/members/50/S000148.jpg Y D 494325 291384
Find:	(.+?)\t(.+?)\t(.+?)\t(.+?)\t(.+?)\t(.+?)\t(.+)
Replace:	<li class="$4 $5">  <div class="name">$1 $2</div>  <img src="$3" alt="mug"> <img src="https://chart.googleapis.com/chart?cht=p&chs=100x80&chds=a&chd=t:$6,$7" alt="E: $6 / I: $7" data-entertainment="$6" data-internet="$7"> </li>
Result:	<li class="Y D">  <div class="name">Barbara Boxer</div>  <img src="beauty-assets/members/50/B000711.jpg" alt="mug"> <img src="https://chart.googleapis.com/chart?cht=p&chs=100x80&chds=a&chd=t:571600,348691" alt="E: 571600 / I: 348691" data-entertainment="571600" data-internet="348691"> </li> <li class="Y D">  <div class="name">Charles Schumer</div>  <img src="beauty-assets/members/50/S000148.jpg" alt="mug"> <img src="https://chart.googleapis.com/chart?cht=p&chs=100x80&chds=a&chd=t:494325,291384" alt="E: 494325 / I: 291384" data-entertainment="494325" data-internet="291384"> </li>

Adding interactivity

This is cake. The hard visualization work is done by jQuery and David DeSandro's isotope library, which we include in the HTML.

The rest is some CSS to make things look nice. And then regular expressions + find-and-replace to get the data in the correct HTML form.

Final result is here.

Bonus Datapoint: When they said Yes or No

While collecting the politicians' positions, I also noted the date of a position change.

With that extra datapoint, creating this static variation of the main graphic took about 2 minutes and was the most popular visualization of the data.

The Big Picture

Visualizations start with the data, so become skilled at finding (research), collecting (web scraping/FOIAing), and cleaning (regular expressions, Google Refine, programming) data.

The best visualization tools are often the ones that are easiest to pick up and use. They should not involve learning a lot of non-transferrable minutiae.

Every tool breaks. So spend time improving your overall data and design skills, and less time worrying about mastering a tool.

Making Data Pretty

Beauty is in the Data, not the Tools

for the International Journalism Festival 2012

Dan Nguyen twitter: @dancow / @propublica
April 27, 2012
Shortlink: http://bit.ly/beautydata

Slide format courtesy of Google's html5slides