1+++
  2title = "Visualizing Cáceres’ OpenData"
  3date = 2020-03-09T00:00:08+00:00
  4updated = 2020-03-19T14:38:41+00:00
  5+++
  6
  7The city of Cáceres has online services to provide [Open Data](http://opendata.caceres.es/) over a wide range of [categories](http://opendata.caceres.es/dataset), all of which are very interesting to explore!
  8
  9We have chosen two different datasets, and will explore four different ways to visualize the data.
 10
 11This post is co-authored with Classmate.
 12
 13## Obtain the data
 14
 15We are interested in the JSON format for the [census in 2017](http://opendata.caceres.es/dataset/informacion-del-padron-de-caceres-2017) and those for the [vias of the city](http://opendata.caceres.es/dataset/vias-urbanas-caceres). This way, we can explore the population and their location in interesting ways! You may follow those two links and select the JSON format under Resources to download it.
 16
 17Why JSON? We will be using [Python](https://python.org/) (3.7 or above) and [matplotlib](https://matplotlib.org/) for quick iteration, and loading the data with [Python’s `json` module](https://docs.python.org/3/library/json.html) will be trivial.
 18
 19## Implementation
 20
 21### Imports and constants
 22
 23We are going to need a lot of things in this code, such as `json` to load the data, `matplotlib` to visualize it, and other data types and type hinting for use in the code.
 24
 25We also want automatic download of the JSON files if they’re missing, so we add their URLs and download paths as constants.
 26
 27```
 28import json
 29import re
 30import os
 31import sys
 32import urllib.request
 33import matplotlib.pyplot as plt
 34from dataclasses import dataclass
 35from collections import namedtuple
 36from datetime import date
 37from pathlib import Path
 38from typing import Optional
 39
 40CENSUS_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionCENSUS&year=2017&format=json'
 41VIAS_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:Via&format=json'
 42
 43CENSUS_JSON = Path('data/demografia/Padrón_Cáceres_2017.json')
 44VIAS_JSON = Path('data/via/Vías_Cáceres.json')
 45```
 46
 47### Data classes
 48
 49[Parse, don’t validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/). By defining a clear data model, we will be able to tell at a glance what information we have available. It will also be typed, so we won’t be confused as to what is what! Python 3.7 introduces `[dataclasses](https://docs.python.org/3/library/dataclasses.html)`, which are a wonderful feature to define… well, data classes concisely.
 50
 51We also have a `[namedtuple](https://docs.python.org/3/library/collections.html#collections.namedtuple)` for points, because it’s extremely common to represent them as tuples.
 52
 53```
 54Point = namedtuple('Point', 'long lat')
 55
 56@dataclass
 57class Census:
 58    year: int
 59    via: int
 60    count_per_year: dict
 61    count_per_city: dict
 62    count_per_gender: dict
 63    count_per_nationality: dict
 64    time_year: int
 65
 66@dataclass
 67class Via:
 68    name: str
 69    kind: str
 70    code: int
 71    history: Optional[str]
 72    old_name: Optional[str]
 73    length: Optional[float]
 74    start: Optional[Point]
 75    middle: Optional[Point]
 76    end: Optional[Point]
 77    geometry: Optional[list]
 78```
 79
 80### Helper methods
 81
 82We will have a little helper method to automatically download the JSON when missing. This is just for convenience, we could as well just download it manually. But it is fun to automate things.
 83
 84```
 85def ensure_file(file, url):
 86    if not file.is_file():
 87        print('Downloading', file.name, 'because it was missing...', end='', flush=True, file=sys.stderr)
 88        file.parent.mkdir(parents=True, exist_ok=True)
 89        urllib.request.urlretrieve(url, file)
 90        print(' Done.', file=sys.stderr)
 91```
 92
 93### Parsing the data
 94
 95I will be honest, parsing Cáceres’ OpenData is a pain in the neck! The official descriptions are huge and not all that helpful. Maybe if one needs documentation for a specific field. But luckily for us, the names are pretty self-descriptive, and we can explore the data to get a feel for what we will find.
 96
 97We define two methods, one to iterate over `Census` values, and another to iterate over `Via` values. Here’s where our friend `[re](https://docs.python.org/3/library/re.html)` comes in, and oh boy the format of the data…
 98
 99For example, the year and via identifier are best extracted from the URI! The information is also available in the `rdfs_label` field, but that’s just a Spanish text! At least the URI will be more reliable… hopefully.
100
101Birth date. They could have used a JSON list, but nah, that would’ve been too simple. Instead, you are given a string separated by semicolons. The values? They could have been dictionaries with names for «year» and «age», but nah! That would’ve been too simple! Instead, you are given strings that look like «2001 (7)», and that’s the year and the count.
102
103The birth place? Sometimes it’s «City (Province) (Count)», but sometimes the province is missing. Gender? Semicolon-separated. And there are only two genders. I know a few people who would be upset just reading this, but it’s not my data, it’s theirs. Oh, and plenty of things are optional. That was a lot of `AttributeError: 'NoneType' object has no attribute 'foo'` to work through!
104
105But as a reward, we have nicely typed data, and we no longer have to deal with this mess when trying to visualize it. For brevity, we will only be showing how to parse the census data, and not the data for the vias. This post is already long enough on its own.
106
107```
108def iter_census(file):
109    with file.open() as fd:
110        data = json.load(fd)
111
112    for row in data['results']['bindings']:
113        year, via = map(int, row['uri']['value'].split('/')[-1].split('-'))
114
115        count_per_year = {}
116        for item in row['schema_birthDate']['value'].split(';'):
117            y, c = map(int, re.match(r'(\d+) \((\d+)\)', item).groups())
118            count_per_year[y] = c
119
120        count_per_city = {}
121        for item in row['schema_birthPlace']['value'].split(';'):
122            match = re.match(r'([^(]+) \(([^)]+)\) \((\d+)\)', item)
123            if match:
124                l, _province, c = match.groups()
125            else:
126                l, c = re.match(r'([^(]+) \((\d+)\)', item).groups()
127
128            count_per_city[l] = int(c)
129
130        count_per_gender = {}
131        for item in row['foaf_gender']['value'].split(';'):
132            g, c = re.match(r'([^(]+) \((\d+)\)', item).groups()
133            count_per_gender[g] = int(c)
134
135        count_per_nationality = {}
136        for item in row['schema_nationality']['value'].split(';'):
137            match = re.match(r'([^(]+) \((\d+)\)', item)
138            if match:
139                g, c = match.groups()
140            else:
141                g, _alt_name, c = re.match(r'([^(]+) \(([^)]+)\) \((\d+)\)', item).groups()
142
143            count_per_nationality[g] = int(c)
144        time_year = int(row['time_year']['value'])
145
146        yield Census(
147            year=year,
148            via=via,
149            count_per_year=count_per_year,
150            count_per_city=count_per_city,
151            count_per_gender=count_per_gender,
152            count_per_nationality=count_per_nationality,
153            time_year=time_year,
154        )
155```
156
157## Visualizing the data
158
159Here comes the fun part! After parsing all the desired data from the mentioned JSON files, we plotted the data in four different graphics making use of Python’s [`matplotlib` library.](https://matplotlib.org/) This powerful library helps with the creation of different visualizations in Python.
160
161### Visualizing the genders in a pie chart
162
163After seeing that there are only two genders in the data of the census, we, displeased, started work in a chart for it. The pie chart was the best option since we wanted to show only the percentages of each gender. The result looks like this:
164
165![](pie_chart.png)
166
167Pretty straight forward, isn’t it? To display this wonderful graphic, we used the following code:
168
169```
170def pie_chart(ax, data):
171    lists = sorted(data.items())
172
173    x, y = zip(*lists)
174    ax.pie(y, labels=x, autopct='%1.1f%%',
175            shadow=True, startangle=90)
176    ax.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.
177```
178
179We pass the axis as the input parameter (later we will explain why) and the data collected from the JSON regarding the genders, which are in a dictionary with the key being the labels and the values the tally of each gender. We sort the data and with some unpacking magic we split it into two values: `x` being the labels and `y` the amount of each gender.
180
181After that we plot the pie chart with the data and labels from `y` and `x`, we specify that we want the percentage with one decimal place with the `autopct` parameter, we enable shadows for the presentation, and specify the start angle at 90º.
182
183### Date tick labels
184
185We wanted to know how many of the living people were born on each year, so we are making a date plot! In the census we have the year each person was born in, and using that information is an easy task after parsing the data (parsing was an important task of this work). The result looks as follows:
186
187![](date_tick.png)
188
189How did we do this? The following code was used:
190
191```
192def date_tick(ax, data):
193    lists = sorted(data.items())
194
195    x, y = zip(*lists)
196    x = [date(year, 1, 1) for year in x]
197    ax.plot(x, y)
198```
199
200Again, we pass in an axis and the data related with the year born, we sort it, split it into two lists, being the keys the years and the values the number per year. After that, we put the years in a date format for the plot to be more accurate. Finally, we plot the values into that wonderful graphic.
201
202### Stacked bar chart
203
204We wanted to know if there was any relation between the latitudes and count per gender, so we developed the following code:
205
206```
207def stacked_bar_chart(ax, data):
208    labels = []
209    males = []
210    females = []
211
212    for latitude, genders in data.items():
213        labels.append(str(latitude))
214        males.append(genders['Male'])
215        females.append(genders['Female'])
216
217    ax.bar(labels, males, label='Males')
218    ax.bar(labels, females, bottom=males, label='Females')
219
220    ax.set_ylabel('Counts')
221    ax.set_xlabel('Latitudes')
222    ax.legend()
223```
224
225The key of the data dictionary is the latitude rounded to two decimals, and value is another dictionary, which is composed by the key that is the name of the gender and the value, the number of people per gender. So, in a single entry of the data dictionary we have the latitude and how many people per gender are in that latitude.
226
227We iterate the dictionary to extract the different latitudes and people per gender (because we know only two genders are used, we hardcode it to two lists). Then we plot them putting the `males` and `females` lists at the bottom and set the labels of each axis. The result is the following:
228
229![](stacked_bar_chart-1.png)
230
231### Scatter plots
232
233This last graphic was very tricky to get right. It’s incredibly hard to find the extent of a city online! We were getting confused because some of the points were way farther than the centre of Cáceres, and the city background is a bit stretched even if the coordinates appear correct. But in the end, we did a pretty good job on it.
234
235```
236def scatter_map(ax, data):
237    xs = []
238    ys = []
239    areas = []
240    for (long, lat), count in data.items():
241        xs.append(long)
242        ys.append(lat)
243        areas.append(count / 100)
244
245    if CACERES_MAP.is_file():
246        ax.imshow(plt.imread(str(CACERES_MAP)), extent=CACERES_EXTENT)
247    else:
248        print('Note:', CACERES_MAP, 'does not exist, not showing it', file=sys.stderr)
249
250    ax.scatter(xs, ys, areas, alpha=0.1)
251```
252
253This time, the keys in the data dictionary are points and the values are the total count of people in that point. We use a normal `for` loop to create the different lists. For the areas on how big the circles we are going to represent will be, we divide the count of people by some number, like `100`, or otherwise they would be huge.
254
255If the file of the map is present, we render it so that we can get a sense on where the points are, but if the file is missing we print a warning.
256
257At last, we draw the scatter plot with some low alpha value (there’s a lot of overlapping points). The result is _absolutely gorgeous_. (For some definitions of gorgeous, anyway):
258
259![](scatter_map.png)
260
261Just for fun, here’s what it looks like if we don’t divide the count by 100 and lower the opacity to `0.01`:
262
263![](scatter_map-2.png)
264
265That’s a big solid blob, and the opacity is only set to `0.01`!
266
267### Drawing all the graphs in the same window
268
269To draw all the graphs in the same window instead of getting four different windows we made use of the [`subplots` function](https://matplotlib.org/3.2.0/api/_as_gen/matplotlib.pyplot.subplots.html), like this:
270
271```
272fig, axes = plt.subplots(2, 2)
273```
274
275This will create a matrix of two by two of axes that we store in the axes variable (fitting name!). Following this code are the different calls to the methods commented before, where we access each individual axis and pass it to the methods to draw on:
276
277```
278pie_chart(axes[0, 0], genders)
279date_tick(axes[0, 1], years)
280stacked_bar_chart(axes[1, 0], latitudes)
281scatter_map(axes[1, 1], positions)
282```
283
284Lastly, we plot the different graphics:
285
286```
287plt.show()
288```
289
290Wrapping everything together, here’s the result:
291
292![](figures-1.png)
293
294The numbers in some of the graphs are a bit crammed together, but we’ll blame that on `matplotlib`.
295
296## Closing words
297
298Wow, that was a long journey! We hope that this post helped you pick some interest in data exploration, it’s such a fun world. We also offer the full download for the code below, because we know it’s quite a bit!
299
300Which of the graphs was your favourite? I personally like the count per date, I think it’s nice to see the growth. Let us know in the comments below!
301
302*download removed*