blog/mdad/visualizing-caceres-opendata/index.html (view raw)
1<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=description content="Official Lonami's website"><meta name=viewport content="width=device-width, initial-scale=1.0, user-scalable=yes"><title> Visualizing Cáceres’ OpenData | Lonami's Blog </title><link rel=stylesheet href=/style.css><body><article><nav class=sections><ul class=left><li><a href=/>lonami's site</a><li><a href=/blog class=selected>blog</a><li><a href=/golb>golb</a></ul><div class=right><a href=https://github.com/LonamiWebs><img src=/img/github.svg alt=github></a><a href=/blog/atom.xml><img src=/img/rss.svg alt=rss></a></div></nav><main><h1 class=title>Visualizing Cáceres’ OpenData</h1><div class=time><p>2020-03-09T00:00:08+00:00<p>last updated 2020-03-19T14:38:41+00:00</div><p>The city of Cáceres has online services to provide <a href=http://opendata.caceres.es/>Open Data</a> over a wide range of <a href=http://opendata.caceres.es/dataset>categories</a>, all of which are very interesting to explore!<p>We have chosen two different datasets, and will explore four different ways to visualize the data.<p>This post is co-authored with Classmate.<h2 id=obtain-the-data>Obtain the data</h2><p>We are interested in the JSON format for the <a href=http://opendata.caceres.es/dataset/informacion-del-padron-de-caceres-2017>census in 2017</a> and those for the <a href=http://opendata.caceres.es/dataset/vias-urbanas-caceres>vias of the city</a>. This way, we can explore the population and their location in interesting ways! You may follow those two links and select the JSON format under Resources to download it.<p>Why JSON? We will be using <a href=https://python.org/>Python</a> (3.7 or above) and <a href=https://matplotlib.org/>matplotlib</a> for quick iteration, and loading the data with <a href=https://docs.python.org/3/library/json.html>Python’s <code>json</code> module</a> will be trivial.<h2 id=implementation>Implementation</h2><h3 id=imports-and-constants>Imports and constants</h3><p>We are going to need a lot of things in this code, such as <code>json</code> to load the data, <code>matplotlib</code> to visualize it, and other data types and type hinting for use in the code.<p>We also want automatic download of the JSON files if they’re missing, so we add their URLs and download paths as constants.<pre><code>import json
2import re
3import os
4import sys
5import urllib.request
6import matplotlib.pyplot as plt
7from dataclasses import dataclass
8from collections import namedtuple
9from datetime import date
10from pathlib import Path
11from typing import Optional
12
13CENSUS_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionCENSUS&year=2017&format=json'
14VIAS_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:Via&format=json'
15
16CENSUS_JSON = Path('data/demografia/Padrón_Cáceres_2017.json')
17VIAS_JSON = Path('data/via/Vías_Cáceres.json')
18</code></pre><h3 id=data-classes>Data classes</h3><p><a href=https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/>Parse, don’t validate</a>. By defining a clear data model, we will be able to tell at a glance what information we have available. It will also be typed, so we won’t be confused as to what is what! Python 3.7 introduces <code>[dataclasses](https://docs.python.org/3/library/dataclasses.html)</code>, which are a wonderful feature to define… well, data classes concisely.<p>We also have a <code>[namedtuple](https://docs.python.org/3/library/collections.html#collections.namedtuple)</code> for points, because it’s extremely common to represent them as tuples.<pre><code>Point = namedtuple('Point', 'long lat')
19
20@dataclass
21class Census:
22 year: int
23 via: int
24 count_per_year: dict
25 count_per_city: dict
26 count_per_gender: dict
27 count_per_nationality: dict
28 time_year: int
29
30@dataclass
31class Via:
32 name: str
33 kind: str
34 code: int
35 history: Optional[str]
36 old_name: Optional[str]
37 length: Optional[float]
38 start: Optional[Point]
39 middle: Optional[Point]
40 end: Optional[Point]
41 geometry: Optional[list]
42</code></pre><h3 id=helper-methods>Helper methods</h3><p>We will have a little helper method to automatically download the JSON when missing. This is just for convenience, we could as well just download it manually. But it is fun to automate things.<pre><code>def ensure_file(file, url):
43 if not file.is_file():
44 print('Downloading', file.name, 'because it was missing...', end='', flush=True, file=sys.stderr)
45 file.parent.mkdir(parents=True, exist_ok=True)
46 urllib.request.urlretrieve(url, file)
47 print(' Done.', file=sys.stderr)
48</code></pre><h3 id=parsing-the-data>Parsing the data</h3><p>I will be honest, parsing Cáceres’ OpenData is a pain in the neck! The official descriptions are huge and not all that helpful. Maybe if one needs documentation for a specific field. But luckily for us, the names are pretty self-descriptive, and we can explore the data to get a feel for what we will find.<p>We define two methods, one to iterate over <code>Census</code> values, and another to iterate over <code>Via</code> values. Here’s where our friend <code>[re](https://docs.python.org/3/library/re.html)</code> comes in, and oh boy the format of the data…<p>For example, the year and via identifier are best extracted from the URI! The information is also available in the <code>rdfs_label</code> field, but that’s just a Spanish text! At least the URI will be more reliable… hopefully.<p>Birth date. They could have used a JSON list, but nah, that would’ve been too simple. Instead, you are given a string separated by semicolons. The values? They could have been dictionaries with names for «year» and «age», but nah! That would’ve been too simple! Instead, you are given strings that look like «2001 (7)», and that’s the year and the count.<p>The birth place? Sometimes it’s «City (Province) (Count)», but sometimes the province is missing. Gender? Semicolon-separated. And there are only two genders. I know a few people who would be upset just reading this, but it’s not my data, it’s theirs. Oh, and plenty of things are optional. That was a lot of <code>AttributeError: 'NoneType' object has no attribute 'foo'</code> to work through!<p>But as a reward, we have nicely typed data, and we no longer have to deal with this mess when trying to visualize it. For brevity, we will only be showing how to parse the census data, and not the data for the vias. This post is already long enough on its own.<pre><code>def iter_census(file):
49 with file.open() as fd:
50 data = json.load(fd)
51
52 for row in data['results']['bindings']:
53 year, via = map(int, row['uri']['value'].split('/')[-1].split('-'))
54
55 count_per_year = {}
56 for item in row['schema_birthDate']['value'].split(';'):
57 y, c = map(int, re.match(r'(\d+) \((\d+)\)', item).groups())
58 count_per_year[y] = c
59
60 count_per_city = {}
61 for item in row['schema_birthPlace']['value'].split(';'):
62 match = re.match(r'([^(]+) \(([^)]+)\) \((\d+)\)', item)
63 if match:
64 l, _province, c = match.groups()
65 else:
66 l, c = re.match(r'([^(]+) \((\d+)\)', item).groups()
67
68 count_per_city[l] = int(c)
69
70 count_per_gender = {}
71 for item in row['foaf_gender']['value'].split(';'):
72 g, c = re.match(r'([^(]+) \((\d+)\)', item).groups()
73 count_per_gender[g] = int(c)
74
75 count_per_nationality = {}
76 for item in row['schema_nationality']['value'].split(';'):
77 match = re.match(r'([^(]+) \((\d+)\)', item)
78 if match:
79 g, c = match.groups()
80 else:
81 g, _alt_name, c = re.match(r'([^(]+) \(([^)]+)\) \((\d+)\)', item).groups()
82
83 count_per_nationality[g] = int(c)
84 time_year = int(row['time_year']['value'])
85
86 yield Census(
87 year=year,
88 via=via,
89 count_per_year=count_per_year,
90 count_per_city=count_per_city,
91 count_per_gender=count_per_gender,
92 count_per_nationality=count_per_nationality,
93 time_year=time_year,
94 )
95</code></pre><h2 id=visualizing-the-data>Visualizing the data</h2><p>Here comes the fun part! After parsing all the desired data from the mentioned JSON files, we plotted the data in four different graphics making use of Python’s <a href=https://matplotlib.org/><code>matplotlib</code> library.</a> This powerful library helps with the creation of different visualizations in Python.<h3 id=visualizing-the-genders-in-a-pie-chart>Visualizing the genders in a pie chart</h3><p>After seeing that there are only two genders in the data of the census, we, displeased, started work in a chart for it. The pie chart was the best option since we wanted to show only the percentages of each gender. The result looks like this:<p><img src=https://lonami.dev/blog/mdad/visualizing-caceres-opendata/pie_chart.png><p>Pretty straight forward, isn’t it? To display this wonderful graphic, we used the following code:<pre><code>def pie_chart(ax, data):
96 lists = sorted(data.items())
97
98 x, y = zip(*lists)
99 ax.pie(y, labels=x, autopct='%1.1f%%',
100 shadow=True, startangle=90)
101 ax.axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle.
102</code></pre><p>We pass the axis as the input parameter (later we will explain why) and the data collected from the JSON regarding the genders, which are in a dictionary with the key being the labels and the values the tally of each gender. We sort the data and with some unpacking magic we split it into two values: <code>x</code> being the labels and <code>y</code> the amount of each gender.<p>After that we plot the pie chart with the data and labels from <code>y</code> and <code>x</code>, we specify that we want the percentage with one decimal place with the <code>autopct</code> parameter, we enable shadows for the presentation, and specify the start angle at 90º.<h3 id=date-tick-labels>Date tick labels</h3><p>We wanted to know how many of the living people were born on each year, so we are making a date plot! In the census we have the year each person was born in, and using that information is an easy task after parsing the data (parsing was an important task of this work). The result looks as follows:<p><img src=https://lonami.dev/blog/mdad/visualizing-caceres-opendata/date_tick.png><p>How did we do this? The following code was used:<pre><code>def date_tick(ax, data):
103 lists = sorted(data.items())
104
105 x, y = zip(*lists)
106 x = [date(year, 1, 1) for year in x]
107 ax.plot(x, y)
108</code></pre><p>Again, we pass in an axis and the data related with the year born, we sort it, split it into two lists, being the keys the years and the values the number per year. After that, we put the years in a date format for the plot to be more accurate. Finally, we plot the values into that wonderful graphic.<h3 id=stacked-bar-chart>Stacked bar chart</h3><p>We wanted to know if there was any relation between the latitudes and count per gender, so we developed the following code:<pre><code>def stacked_bar_chart(ax, data):
109 labels = []
110 males = []
111 females = []
112
113 for latitude, genders in data.items():
114 labels.append(str(latitude))
115 males.append(genders['Male'])
116 females.append(genders['Female'])
117
118 ax.bar(labels, males, label='Males')
119 ax.bar(labels, females, bottom=males, label='Females')
120
121 ax.set_ylabel('Counts')
122 ax.set_xlabel('Latitudes')
123 ax.legend()
124</code></pre><p>The key of the data dictionary is the latitude rounded to two decimals, and value is another dictionary, which is composed by the key that is the name of the gender and the value, the number of people per gender. So, in a single entry of the data dictionary we have the latitude and how many people per gender are in that latitude.<p>We iterate the dictionary to extract the different latitudes and people per gender (because we know only two genders are used, we hardcode it to two lists). Then we plot them putting the <code>males</code> and <code>females</code> lists at the bottom and set the labels of each axis. The result is the following:<p><img src=https://lonami.dev/blog/mdad/visualizing-caceres-opendata/stacked_bar_chart-1.png><h3 id=scatter-plots>Scatter plots</h3><p>This last graphic was very tricky to get right. It’s incredibly hard to find the extent of a city online! We were getting confused because some of the points were way farther than the centre of Cáceres, and the city background is a bit stretched even if the coordinates appear correct. But in the end, we did a pretty good job on it.<pre><code>def scatter_map(ax, data):
125 xs = []
126 ys = []
127 areas = []
128 for (long, lat), count in data.items():
129 xs.append(long)
130 ys.append(lat)
131 areas.append(count / 100)
132
133 if CACERES_MAP.is_file():
134 ax.imshow(plt.imread(str(CACERES_MAP)), extent=CACERES_EXTENT)
135 else:
136 print('Note:', CACERES_MAP, 'does not exist, not showing it', file=sys.stderr)
137
138 ax.scatter(xs, ys, areas, alpha=0.1)
139</code></pre><p>This time, the keys in the data dictionary are points and the values are the total count of people in that point. We use a normal <code>for</code> loop to create the different lists. For the areas on how big the circles we are going to represent will be, we divide the count of people by some number, like <code>100</code>, or otherwise they would be huge.<p>If the file of the map is present, we render it so that we can get a sense on where the points are, but if the file is missing we print a warning.<p>At last, we draw the scatter plot with some low alpha value (there’s a lot of overlapping points). The result is <em>absolutely gorgeous</em>. (For some definitions of gorgeous, anyway):<p><img src=https://lonami.dev/blog/mdad/visualizing-caceres-opendata/scatter_map.png><p>Just for fun, here’s what it looks like if we don’t divide the count by 100 and lower the opacity to <code>0.01</code>:<p><img src=https://lonami.dev/blog/mdad/visualizing-caceres-opendata/scatter_map-2.png><p>That’s a big solid blob, and the opacity is only set to <code>0.01</code>!<h3 id=drawing-all-the-graphs-in-the-same-window>Drawing all the graphs in the same window</h3><p>To draw all the graphs in the same window instead of getting four different windows we made use of the <a href=https://matplotlib.org/3.2.0/api/_as_gen/matplotlib.pyplot.subplots.html><code>subplots</code> function</a>, like this:<pre><code>fig, axes = plt.subplots(2, 2)
140</code></pre><p>This will create a matrix of two by two of axes that we store in the axes variable (fitting name!). Following this code are the different calls to the methods commented before, where we access each individual axis and pass it to the methods to draw on:<pre><code>pie_chart(axes[0, 0], genders)
141date_tick(axes[0, 1], years)
142stacked_bar_chart(axes[1, 0], latitudes)
143scatter_map(axes[1, 1], positions)
144</code></pre><p>Lastly, we plot the different graphics:<pre><code>plt.show()
145</code></pre><p>Wrapping everything together, here’s the result:<p><img src=https://lonami.dev/blog/mdad/visualizing-caceres-opendata/figures-1.png><p>The numbers in some of the graphs are a bit crammed together, but we’ll blame that on <code>matplotlib</code>.<h2 id=closing-words>Closing words</h2><p>Wow, that was a long journey! We hope that this post helped you pick some interest in data exploration, it’s such a fun world. We also offer the full download for the code below, because we know it’s quite a bit!<p>Which of the graphs was your favourite? I personally like the count per date, I think it’s nice to see the growth. Let us know in the comments below!<p><em>download removed</em></main><footer><div><p>Share your thoughts, or simply come hang with me <a href=https://t.me/LonamiWebs><img src=/img/telegram.svg alt=Telegram></a> <a href=mailto:totufals@hotmail.com><img src=/img/mail.svg alt=Mail></a></div></footer></article><p class=abyss>Glaze into the abyss… Oh hi there!