pagong

Data Mining and Data Warehousing

2020-08-23T22:00:00+00:00

Data Mining and Data Warehousing

¶Data Mining and Data Warehousing

2020-08-24

During 2020 at university, this subject ("Minería de Datos y Almacenes de Datos") had us write blog posts as assignments. I think it would be really fun and I wanted to preserve that work here, with the hopes it's interesting to someone.

The posts were auto-generated from the original HTML files and manually anonymized later.

Privado: Final NoSQL evaluation

2020-05-13T22:00:00+00:00

Privado: Final NoSQL evaluation

This evaluation is a bit different to my previous one because this time I have been tasked to evaluate student a(i - 2), and because I am i = 11 that happens to be a(9) = a classmate.

Created 2020-05-13
Modified 2020-05-14

¶Classmate’s Evaluation

Grading: A.

The post I have evaluated is Trabajo en grupo – Bases de datos NoSQL, 3ª entrada: Aplicación con una Base de datos NoSQL seleccionada.

It starts with a very brief introduction with who has written the post, what data they will be using, and what database they have chosen.

They properly describe their objective, how they will do it and what library will be used.

They also explain where they obtain the data from, and what other things the site can do, which is a nice bonus.

The post continues listing and briefly explaining all the tools used and what they are for, including commands to execute.

At last, they list what files their project uses, what they do, and contains a showcase of images which lets the reader know what the application does.

All in all, in my opinion, it’s clear they have put work into this entry and I have not noticed any major flaws, so they deserve the highest grade.

A practical example with Hadoop

2020-04-17T22:00:00+00:00

A practical example with Hadoop

In our previous Hadoop post, we learnt what it is, how it originated, and how it works, from a theoretical standpoint. Here we will instead focus on a more practical example with Hadoop.

Created 2020-03-30
Modified 2020-04-18

This post will reproduce the example on Chapter 2 of the book Hadoop: The Definitive Guide, Fourth Edition (pdf,code), that is, finding the maximum global-wide temperature for a given year.

¶Installation

Before running any piece of software, its executable code must first be downloaded into our computers so that we can run it. Head over to Apache Hadoop’s releases and download the latest binary version at the time of writing (3.2.1).

We will be using the Linux Mint distribution because I love its simplicity, although the process shown here should work just fine on any similar Linux distribution such as Ubuntu.

Once the archive download is complete, extract it with any tool of your choice (graphical or using the terminal) and execute it. Make sure you have a version of Java installed, such as OpenJDK.

Here are all the three steps in the command line:

wget https://apache.brunneis.com/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
tar xf hadoop-3.2.1.tar.gz
hadoop-3.2.1/bin/hadoop version

We will be using the two example data files that they provide in their GitHub repository, although the full dataset is offered by the National Climatic Data Center (NCDC).

We will also unzip and concatenate both files into a single text file, to make it easier to work with. As a single command pipeline:

curl https://raw.githubusercontent.com/tomwhite/hadoop-book/master/input/ncdc/all/190{1,2}.gz | gunzip > 190x

This should create a 190x text file in the current directory, which will be our input data.

¶Processing data

To take advantage of Hadoop, we have to design our code to work in the MapReduce model. Both the map and reduce phase work on key-value pairs as input and output, and both have a programmer-defined function.

We will use Java, because it’s a dependency that we already have anyway, so might as well.

Our map function needs to extract the year and air temperature, which will prepare the data for later use (finding the maximum temperature for each year). We will also drop bad records here (if the temperature is missing, suspect or erroneous).

Copy or reproduce the following code in a file called MaxTempMapper.java, using any text editor of your choice:

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class MaxTempMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private static final int TEMP_MISSING = 9999;
    private static final String GOOD_QUALITY_RE = "[01459]";

    @Override
    public void map(LongWritable key, Text value, Context context)
            throws IOException, InterruptedException {
        String line = value.toString();
        String year = line.substring(15, 19);
        String temp = line.substring(87, 92).replaceAll("^\\+", "");
        String quality = line.substring(92, 93);

        int airTemperature = Integer.parseInt(temp);
        if (airTemperature != TEMP_MISSING && quality.matches(GOOD_QUALITY_RE)) {
            context.write(new Text(year), new IntWritable(airTemperature));
        }
    }
}

Now, let’s create the MaxTempReducer.java file. Its job is to reduce the data from multiple values into just one. We do that by keeping the maximum out of all the values we receive:

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class MaxTempReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context)
            throws IOException, InterruptedException {
        Iterator<IntWritable> iter = values.iterator();
        if (iter.hasNext()) {
            int maxValue = iter.next().get();
            while (iter.hasNext()) {
                maxValue = Math.max(maxValue, iter.next().get());
            }
            context.write(key, new IntWritable(maxValue));
        }
    }
}

Except for some Java weirdness (…why can’t we just iterate over an Iterator? Or why can’t we just manually call next() on an Iterable?), our code is correct. There can’t be a maximum if there are no elements, and we want to avoid dummy values such as Integer.MIN_VALUE.

We can also take a moment to appreciate how absolutely tiny this code is, and it’s Java! Hadoop’s API is really awesome and lets us write such concise code to achieve what we need.

Last, let’s write the main method, or else we won’t be able to run it. In our new file MaxTemp.java:

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class MaxTemp {
    public static void main(String[] args) throws Exception {
        if (args.length != 2) {
            System.err.println("usage: java MaxTemp <input path> <output path>");
            System.exit(-1);
        }

        Job job = Job.getInstance();

        job.setJobName("Max temperature");
        job.setJarByClass(MaxTemp.class);
        job.setMapperClass(MaxTempMapper.class);
        job.setReducerClass(MaxTempReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        boolean result = job.waitForCompletion(true);

        System.exit(result ? 0 : 1);
    }
}

And compile by including the required .jar dependencies in Java’s classpath with the -cp switch:

javac -cp "hadoop-3.2.1/share/hadoop/common/*:hadoop-3.2.1/share/hadoop/mapreduce/*" *.java

At last, we can run it (also specifying the dependencies in the classpath, this one’s a mouthful):

java -cp ".:hadoop-3.2.1/share/hadoop/common/*:hadoop-3.2.1/share/hadoop/common/lib/*:hadoop-3.2.1/share/hadoop/mapreduce/*:hadoop-3.2.1/share/hadoop/mapreduce/lib/*:hadoop-3.2.1/share/hadoop/yarn/*:hadoop-3.2.1/share/hadoop/yarn/lib/*:hadoop-3.2.1/share/hadoop/hdfs/*:hadoop-3.2.1/share/hadoop/hdfs/lib/*" MaxTemp 190x results

Hooray! We should have a new results/ folder along with the following files:

$ ls results
part-r-00000  _SUCCESS
$ cat results/part-r-00000 
1901	317
1902	244

It worked! Now this example was obviously tiny, but hopefully enough to demonstrate how to get the basics running on real world data.

Developing a Python application for Cassandra

2020-04-15T22:00:00+00:00

Developing a Python application for Cassandra

Warning: this post is, in fact, a shameless self-plug to my own library. If you continue reading, you accept that you are okay with this. Otherwise, please close the tab, shut down your computer, and set it on fire.__(Also, that was a joke. Please don’t do that.)

Created 2020-03-23
Modified 2020-04-16

Let’s do some programming! Today we will be making a tiny CLI application in Python that queries Telegram’s API and stores the data in Cassandra.

¶Our goal

Our goal is to make a Python console application. This application will connect to Telegram, and ask for your account credentials. Once you have logged in, the application will fetch all of your open conversations and we will store these in Cassandra.

With the data saved in Cassandra, we can now very efficiently query information about your conversations given their identifier offline (no need to query Telegram anymore).

In short, we are making an application that performs efficient offline queries to Cassandra to print out information about your Telegram conversations given the ID you want to query.

¶Data model

The application itself is really simple, and we only need one table to store all the relevant information we will be needing. This table called **users** will contain the following columns:

**id**, of type int. This will also be the primary key and we’ll use it to query the database later on.
**first_name**, of type varchar. This field contains the first name of the stored user.
**last_name**, of type varchar. This field contains the last name of the stored user.
**username**, of type varchar. This field contains the username of the stored user. Because Cassandra uses a wide column storage model, direct access through a key is the most efficient way to query the database. In our case, the key is the primary key of the users table, using the id column. The index for the primary key is ready to be used as soon as we create the table, so we don’t need to create it on our own.

¶Dependencies

Because we will program it in Python, you need Python installed. You can install it using a package manager of your choice or heading over to the Python downloads section, but if you’re on Linux, chances are you have it installed already.

Once Python 3.5 or above is installed, get a copy of the Cassandra driver for Python and Telethon through pip:

pip install cassandra-driver telethon

For more details on that, see the installation guide for cassandra-driver, or the installation guide for telethon.

As we did in our previous post, we will setup a new keyspace for this application with cqlsh. We will also create a table to store the users into. This could all be automated in the Python code, but because it’s a one-time thing, we prefer to use cqlsh.

Make sure that Cassandra is running in the background. We can’t make queries to it if it’s not running.

$ bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.6 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> create keyspace mdad with replication = {'class': 'SimpleStrategy', 'replication_factor': 3};
cqlsh> use mdad;
cqlsh:mdad> create table users(id int primary key, first_name varchar, last_name varchar, username varchar);

Python installed? Check. Python dependencies? Check. Cassandra ready? Check.

¶The code

¶Getting users

The first step is connecting to Telegram’s API, for which we’ll use Telethon, a wonderful (wink, wink) Python library to interface with it.

As with most APIs, we need to supply our API key in order to use it (here API_ID and API_HASH). We will refer to them as constants. At the end, you may download the entire code and use my own key for this example. But please don’t use those values for your other applications!

It’s pretty simple: we create a client, and for every dialog (that is, open conversation) we have, do some checks:

If it’s an user, we just store that in a dictionary mapping ID → User.
Else if it’s a group, we iterate over the participants and store those users instead.

async def load_users():
    from telethon import TelegramClient

    users = {}

    async with TelegramClient(SESSION, API_ID, API_HASH) as client:
        async for dialog in client.iter_dialogs():
            if dialog.is_user:
                user = dialog.entity
                users[user.id] = user
                print('found user:', user.id, file=sys.stderr)

            elif dialog.is_group:
                async for user in client.iter_participants(dialog):
                    users[user.id] = user
                    print('found member:', user.id, file=sys.stderr)

    return list(users.values())

With this we have a mapping ID to user, so we know we won’t have duplicates. We simply return the list of user values, because that’s all we care about.

¶Saving users

Inserting users into Cassandra is pretty straightforward. We take the list of User objects as input, and prepare a new INSERT statement that we can reuse (because we will be using it in a loop, this is the best way to do it).

For each user, execute the statement with the user data as input parameters. Simple as that.

def save_users(session, users):
    insert_stmt = session.prepare(
        'INSERT INTO users (id, first_name, last_name, username) ' 
        'VALUES (?, ?, ?, ?)')

    for user in users:
        row = (user.id, user.first_name, user.last_name, user.username)
        session.execute(insert_stmt, row)

¶Fetching users

Given a list of users, yield all of them from the database. Similar to before, we prepare a SELECT statement and just execute it repeatedly over the input user IDs.

def fetch_users(session, users):
    select_stmt = session.prepare('SELECT * FROM users WHERE id = ?')

    for user_id in users:
        yield session.execute(select_stmt, (user_id,)).one()

¶Parsing arguments

We’ll be making a little CLI application, so we need to parse console arguments. It won’t be anything fancy, though. For that we’ll be using Python’s argparse module:

def parse_args():
    import argparse

    parser = argparse.ArgumentParser(
        description='Dump and query Telegram users')

    parser.add_argument('users', type=int, nargs='*',
        help='one or more user IDs to query for')

    parser.add_argument('--load-users', action='store_true',
        help='load users from Telegram (do this first run)')

    return parser.parse_args()

¶All together

Last, the entry point. We import a Cassandra Cluster, and connect to some default keyspace (we called it mdad earlier).

If the user wants to load the users into the database, we’ll do just that first.

Then, for each user we fetch from the database, we print it. Last names and usernames are optional, so don’t print those if they’re missing (None).

async def main(args):
    from cassandra.cluster import Cluster

    cluster = Cluster(CLUSTER_NODES)
    session = cluster.connect(KEYSPACE)

    if args.load_users:
        users = await load_users()
        save_users(session, users)

    for user in fetch_users(session, args.users):
        print('User', user.id, ':')
        print('  First name:', user.first_name)
        if user.last_name:
            print('  Last name:', user.last_name)
        if user.username:
            print('  Username:', user.username)

        print()

if __name__ == '__main__':
    asyncio.run(main(parse_args()))

Because Telethon is an [asyncio](https://docs.python.org/3/library/asyncio.html) library, we define it as async def main(...) and run it with asyncio.run(main(...)).

Here’s what it looks like in action:

$ python data.py --help
usage: data.py [-h] [--load-users] [users [users ...]]

Dump and query Telegram users

positional arguments:
  users         one or more user IDs to query for

optional arguments:
  -h, --help    show this help message and exit
  --load-users  load users from Telegram (do this first run)

$ python data.py --load-users
found user: 487158
found member: 59794114
found member: 487158
found member: 191045991
(...a lot more output)

$ python data.py 487158 59794114
User 487158 :
  First name: Rick
  Last name: Pickle

User 59794114 :
  Firt name: Peter
  Username: pete

Telegram’s data now persists in Cassandra, and we can efficiently query it whenever we need to! I would’ve shown a video presenting its usage, but I’m afraid that would leak some of the data I want to keep private :-).

Feel free to download the code and try it yourself:

download removed

¶References

Introduction to Hadoop and its MapReduce

2020-03-31T22:00:00+00:00

Introduction to Hadoop and its MapReduce

Hadoop is an open-source, free, Java-based programming framework that helps processing large datasets in a distributed environment and the problems that arise when trying to harness the knowledge from BigData, capable of running on thousands of nodes and dealing with petabytes of data. It is based on Google File System (GFS) and originated from the work on the Nutch open-source project on search engines.

Created 2020-03-30
Modified 2020-04-01

Hadoop also offers a distributed filesystem (HDFS) enabling for fast transfer among nodes, and a way to program with MapReduce.

It aims to strive for the 4 V’s: Volume, Variety, Veracity and Velocity. For veracity, it is a secure environment that can be trusted.

¶Milestones

The creators of Hadoop are Doug Cutting and Mike Cafarella, who just wanted to design a search engine, Nutch, and quickly found the problems of dealing with large amounts of data. They found their solution with the papers Google published.

The name comes from the plush of Cutting’s child, a yellow elephant.

In July 2005, Nutch used GFS to perform MapReduce operations.
In February 2006, Nutch started a Lucene subproject which led to Hadoop.
In April 2007, Yahoo used Hadoop in a 1 000-node cluster.
In January 2008, Apache took over and made Hadoop a top-level project.
In July 2008, Apache tested a 4000-node cluster. The performance was the fastest compared to other technologies that year.
In May 2009, Hadoop sorted a petabyte of data in 17 hours.
In December 2011, Hadoop reached 1.0.
In May 2012, Hadoop 2.0 was released with the addition of YARN (Yet Another Resource Navigator) on top of HDFS, splitting MapReduce and other processes into separate components, greatly improving the fault tolerance.

From here onwards, many other alternatives have born, like Spark, Hive & Drill, Kafka, HBase, built around the Hadoop ecosystem.

As of 2017, Amazon has clusters between 1 and 100 nodes, Yahoo has over 100 000 CPUs running Hadoop, AOL has clusters with 50 machines, and Facebook has a 320-machine (2 560 cores) and 1.3PB of raw storage.

¶Why not use RDBMS?

Relational database management systems simply cannot scale horizontally, and vertical scaling will require very expensive servers. Similar to RDBMS, Hadoop has a notion of jobs (analogous to transactions), but without ACID or concurrency control. Hadoop supports any form of data (unstructured or semi-structured) in read-only mode, and failures are common but there’s a simple yet efficient fault tolerance.

So what problems does Hadoop solve? It solves the way we should think about problems, and distributing them, which is key to do anything related with BigData nowadays. We start working with clusters of nodes, and coordinating the jobs between them. Hadoop’s API makes this really easy.

Hadoop also takes very seriously the loss of data with replication, and if a node falls, they are moved to a different node.

¶Major components

The previously-mentioned HDFS runs on commodity machine, which are cost-friendly. It is very fault-tolerant and efficient enough to process huge amounts of data, because it splits large files into smaller chunks (or blocks) that can be more easily handled. Multiple nodes can work on multiple chunks at the same time.

NameNode stores the metadata of the various datablocks (map of blocks) along with their location. It is the brain and the master in Hadoop’s master-slave architecture, also known as the namespace, and makes use of the DataNode.

A secondary NameNode is a replica that can be used if the first NameNode dies, so that Hadoop doesn’t shutdown and can restart.

DataNode stores the blocks of data, and are the slaves in the architecture. This data is split into one or more files. Their only job is to manage this access to the data. They are often distributed among racks to avoid data lose.

JobTracker creates and schedules jobs from the clients for either map or reduce operations.

TaskTracker runs MapReduce tasks assigned to the current data node.

When clients need data, they first interact with the NameNode and replies with the location of the data in the correct DataNode. Client proceeds with interaction with the DataNode.

¶MapReduce

MapReduce, as the name implies, is split into two steps: the map and the reduce. The map stage is the «divide and conquer» strategy, while the reduce part is about combining and reducing the results.

The mapper has to process the input data (normally a file or directory), commonly line-by-line, and produce one or more outputs. The reducer uses all the results from the mapper as its input to produce a new output file itself.

When reading the data, some may be junk that we can choose to ignore. If it is valid data, however, we label it with a particular type that can be useful for the upcoming process. Hadoop is responsible for splitting the data accross the many nodes available to execute this process in parallel.

There is another part to MapReduce, known as the Shuffle-and-Sort. In this part, types or categories from one node get moved to a different node. This happens with all nodes, so that every node can work on a complete category. These categories are known as «keys», and allows Hadoop to scale linearly.

¶References

Data Warehousing and OLAP

2020-03-31T22:00:00+00:00

Data Warehousing and OLAP

Business intelligence (BI) refers to systems used to gain insights from data, traditionally taken from relational databases and being used to build a data warehouse. Performance and scalability are key aspects of BI systems.

Created 2020-03-23
Modified 2020-04-01

Commonly, the data in the warehouse is a transformation of the original, operational data into a form better suited for reporting and analysis.

This whole process is known as Online Analytical Processing (OLAP), and is different to the approach taken by relational databases, which is known as Online Transaction Processing (OLTP) and is optimized for individual transactions. OLAP is based on multidimensional databases simply by the way it works.

The Business Intelligence Semantic Model (BISM) refers to the different semantics in which data can be accessed and queried.

On the one hand, MDX is the language used for Microsoft’s BISM of multidimensional mode, and on the other, DAX is the language of tabular mode, based on Excel’s formula language and designed to be easy to use by those familiar with Excel.

¶Types of data

The business data is often called detail data or fact data, goes in a de-normalized table called the fact table. The term «facts» literally refers to the facts, such as number of products sold and amount received for products sold. Different tables will often represent different dimensions of the data, where «dimensions» simply means different ways to look at the data.

Data can also be referred to as measures, because most of it is numbers and subject to aggregations. By measures, we refer to these values and numbers.

Multidimensional databases are formed with separate fact and dimension tables, grouped to create a «cube» with both facts and dimensions.

¶Places to store data

Three different terms are often heard when talking about the places where data is stored: data lakes, data warehouses, and data marts. All of these have different target users, cost, size and growth.

The data lake contains all the data generated by your business. Nothing is filtered out, not even cancelled or invalid transactions. If there are future plans to use the data, or a need to analyze it in various ways, a data lake is often necessary.

The data warehouse contains structured data, or has already been modelled. It’s also multi-purpose, but often of a lot smaller scale. Operational users are able to easily evaluate reports or analyze performance here, since it is built for their needs.

The data mart contains a small portion of the data, and is often part of data warehouses themselves. It can be seen as a subsection built for specific departments, and as a benefit, users get isolated security and performance. The data here is clean, and subject-oriented.

¶Ways to store data

Data is often stored de-normalized, because it would not be feasible to store otherwise.

There are two main techniques to implement data warehouses, known as Inmon approach and Kimball approach. They are named after Ralph Kimball et al. for their work on «The Data Warehouse Lifecycle Toolkit», and Bill Inmon et al. for their work on «Corporate Information Factory» respectively.

When several independent systems identify and store data in different ways, we face what’s known as the problem of the stovepipe. Something as simple as trying to connect these systems or use their data in a warehouse results in an overly complicated system.

To tackle this issue, Kimball advocates the use of «conformed dimensions», that is, some dimensions will be «of interest», and have the same attributes and rollups (or at least a subset) in different data marts. This way, warehouses contain dimensional databases to ease analysis in the data marts it is composed of, and users query the warehouse.

The Inmon approach on the other hand has the warehouse laid out in third normal form, and users query the data marts, not the warehouse (so the data marts are dimensional in nature).

¶Key takeaways

«BI» stands for «Business Intelligence» and refers to the system that perform data analysis.
«BISM» stands for «Business Intelligence Semantic Model», and Microsoft has two languages to query data: MDX and DAX.
«OLAP» stands for «Online Analytical Processing», and «OLTP» for «Online Transaction Processing».
Data mart, warehouse and lake refer to places at different scales and with different needs to store data.
Inmon and Kimbal are different ways to implement data warehouses.
Data facts contains various measures arranged into different dimensions, which together form a data cube.

¶References

Chapter 1 – Professional Microsoft SQL Server 2012 Analysis Services with MDX and DAX (Harinath et al., 2012)
YouTube – Data Mining in SQL Server Analysis Services
Almacenes de Datos y Procesamiento Analítico On-Line (Félix R.)
YouTube – What are Dimensions and Measures?
Data Lake vs Data Warehouse vs Data Mart

Cassandra: Introducción

2020-03-29T22:00:00+00:00

Cassandra: Introducción

Created 2020-03-05
Modified 2020-03-30

Este es el primer post en la serie sobre Cassandra, en el cuál introduciremos dicha bases de datos NoSQL y veremos sus características e instalación.

Otros posts en esta serie:

Cassandra: Introducción (este post)
Cassandra: Operaciones Básicas y Arquitectura

Este post está hecho en colaboración con un compañero.

¶Finalidad de la tecnología

Apache Cassandra es una base de datos NoSQL distribuida y de código abierto (con un espejo en GitHub). Su filosofía es de tipo «clave-valor», y puede manejar grandes volúmenes de datos

Entre sus objetivos, busca ser escalable horizontalmente (puede replicarse en varios centros manteniendo la latencia baja) y alta disponibilidad sin ceder en rendimiento.

¶Cómo funciona

Instancias de Cassandra se distribuyen en nodos iguales (es decir, no hay maestro-esclavo) que se comunican entre sí (P2P). De este modo, da buen soporte entre varios centros de datos, con redundancia y réplicas síncronas.

Con respecto al modelo de datos, Cassandra particiona las filas con el objetivo de re-organizarla a lo largo distintas tablas. Como clave primaria, se usa un primer componente conocido como «clave de la partición». Dentro de cada partición, las filas se agrupan según el resto de columnas de la clave. Cualquier otra columna se puede indexar independientemente de la clave primaria.

Las tablas se pueden crear, borrar, actualizar y consultar sin bloqueos. No hay soporte para JOIN o subconsultas, pero Cassandra prefiere de-normalizar los datos haciendo uso de características como coleciones.

Para realizar las operaciones sobre cassandra se usa CQL (Cassandra Query Language), que tiene una sintaxis muy similar a SQL.

¶Características

Como ya hemos mencionado antes, la arquitectura de Cassandra es decentralizada. No tiene un único punto que pudiera fallar porque todos los nodos son iguales (sin maestros), y por lo tanto, cualquiera puede dar servicio a la petición.

Los datos se encuentran replicados entre los distintos nodos del clúster (lo que ofrece gran tolerancia a fallos sin necesidad de interrumpir la aplicación), y es trivial escalar añadiendo más nodos al sistema.

El nivel de consistencia para lecturas y escrituras es configurable.

Siendo de la familia Apache, Cassandra ofrece integración con Apache Hadoop para tener soporte MapReduce.

¶Arista dentro del Teorema CAP

Cassandra se encuentra dentro de la esquina «AP» junto con CouchDB y otros, porque garantiza tanto la disponibilidad como la tolerancia a fallos.

Sin embargo, puede configurarse como un sistema «CP» si se prefiere respetar la consistencia en todo momento.

¶Descarga

Se pueden seguir las instrucciones de la página oficial para descargar Cassandra. Para ello, se debe clicar en la última versión para descargar el archivo. En nuestro caso, esto es el enlace nombrado «3.11.6», versión que utilizamos.

¶Instalación

Cassandra no ofrece binarios para Windows, por lo que usaremos Linux para instalarlo. En nuestro caso, tenemos un sistema Linux Mint (derivado de Ubuntu), pero una máquina virtual con cualquier Linux debería funcionar.

Debemos asegurarnos de tener Java y Python 2 instalado mediante el siguiente comando:

apt install openjdk-8-jdk openjdk-8-jre python2.7

Para verificar que la instalación ha sido correcta, podemos mostrar las versiones de los programas:

$ java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

$ python2 --version
Python 2.7.17

Una vez las dependencias estén instaladas, extraemos el fichero descargado o bien mediante la interfaz gráfica de nuestro sistema, o bien mediante un comando:

tar xf apache-cassandra-3.11.6-bin.tar.gz

Y finalmente, lanzar la ejecución de Cassandra:

apache-cassandra-3.11.6/bin/cassandra

Es posible que tarde un poco en abrirse, pero luego debería haber muchas líneas de log indicando. Para apagar el servidor, simplemente basta con pulsar Ctrl+C.

¶Referencias

Privado: NoSQL evaluation

2020-03-27T23:00:00+00:00

Privado: NoSQL evaluation

This evaluation is based on the criteria for the first delivery described by Trabajos en grupo sobre Bases de Datos NoSQL.

Created 2020-03-16
Modified 2020-03-28

I have chosen to evaluate the following people and works:

a12: Classmate (username) with Druid.
a21: Classmate (username) with Neo4J.

¶Classmate’s Evaluation

Grading: A.

The post evaluated is Bases de datos NoSQL – Apache Druid – Primera entrega.

It is a very well-written, complete post, with each section meeting one of the points in the required criteria. The only thing that bothered me a little is the abuse of strong emphasis in the text, which I found quite distracting. However, the content deserves the highest grading.

¶Classmate’s Evaluation

Grading: A.

The post evaluated is Bases de datos NoSQL – Neo4j – Primera entrega.

Well-written post, although a bit smaller than Classmate’s, but that’s not really an issue. It still talks about everything it should talk and includes photos to go along the text which help. There is no noticeable wrong things in it, so it gets the highest grading as well.

Mining of Massive Datasets

2020-03-27T23:00:00+00:00

Mining of Massive Datasets

In this post we will talk about the Chapter 1 of the book Mining of Massive Datasets Leskovec, J. et al., available online, and I will summarize and share my thoughts on it.

Created 2020-03-16
Modified 2020-03-28

Data mining often refers to the discovery of models for data, where the model can be for statistics, machine learning, summarizing, extracting features, or other computational approaches to perform complex queries on the data.

Commonly, problems related to data mining involve discovering unusual events hidden in massive data sets. There is another problem when trying to achieve Total Information Awareness (TIA), though, a project that was proposed by the Bush administration but shut down. The problem is, if you look at so much data, and try to find activities that look like (for example) terrorist behavior, inevitably one will find other illicit activities that are not terrorism with bad consequences. So it is important to narrow the activities we are looking for, in this case.

When looking at data, even completely random data, for a certain event type, the event will likely occur. With more data, it will occur more times. However, these are bogus results. The Bonferroni correction gives a statistically sound way to avoid most of these bogus results, however, the Bonferroni’s Principle can be used as an informal version to achieve the same thing.

For that, we calculate the expected number of occurrences of the events you are looking for on the assumption that data is random. If this number is way larger than the number of real instances one hoped to find, then nearly everything will be Bogus.

When analysing documents, some words will be more important than others, and can help determine the topic of the document. One could think the most repeated words are the most important, but that’s far from the truth. The most common words are the stop-words, which carry no meaning, reason why we should remove them prior to processing. We are mostly looking for rare nouns.

There are of course formal measures for how concentrated into relatively few documents are the occurrences of a given word, known as TF.IDF (Term Frequency times In-verse Document Frequency). We won’t go into details on how to compute it, because there are multiple ways.

Hash functions are also frequently used, because they can turn hash keys into a bucket number (the index of the bucket where this hash key belongs). They «randomize» and spread the universe of keys into a smaller number of buckets, useful for storage and access.

An index is an efficient structure to query for values given a key, and can be built with hash functions and buckets.

Having all of these is important when analysing documents when doing data mining, because otherwise it would take far too long.

MongoDB: Operaciones Básicas y Arquitectura

2020-03-19T23:00:00+00:00

MongoDB: Operaciones Básicas y Arquitectura

Este es el segundo post en la serie sobre MongoDB, con una breve descripción de las operaciones básicas (tales como inserción, recuperación e indexado), y ejecución por completo junto con el modelo de datos y arquitectura.

Created 2020-03-05
Modified 2020-03-20

Otros posts en esta serie:

MongoDB: Introducción
MongoDB: Operaciones Básicas y Arquitectura (este post)

Este post está hecho en colaboración con un compañero, y en él veremos algunos ejemplos de las operaciones básicas (CRUD) sobre MongoDB.

Empezaremos viendo cómo creamos una nueva base de datos dentro de MongoDB y una nueva colección donde poder insertar nuestros documentos.

¶Creación de una base de datos e inserción de un primer documento

Podemos ver las bases de datos que tenemos disponibles ejecutando el comando:

> show databases
admin   0.000GB
config  0.000GB
local   0.000GB

Para crear una nueva base de datos, o utilizar una de las que tenemos creadas ejecutamos use junto con el nombre que le vamos a dar:

> use new_DB
switched to db new_DB

Una vez hecho esto, podemos ver que si volvemos a ejecutar «show databases», la nueva base de datos no aparece. Esto es porque para que Mongo registre una base de datos en la lista de las existentes, necesitamos insertar al menos un nuevo documento en una colección de esta. Lo podemos hacer de la siguiente forma:

> db.movie.insert({"name":"tutorials point"})
WriteResult({ "nInserted" : 1 })

> show databases
admin       0.000GB
config      0.000GB
local       0.000GB
movie       0.000GB

Al igual que podemos ver las bases de datos existentes, también podemos consultar las colecciones que existen dentro de estas. Siguiendo la anterior ejecución, si ejecutamos:

> show collections
movie

¶Borrar base de datos

Para borrar una base de datos tenemos que ejecutar el siguiente comando:

> db.dropDatabase()
{ "dropped" : "new_DB", "ok" : 1 }

¶Crear colección

Para crear una colección podemos hacerlo de dos formas. O bien mediante el comando:

db.createCollection(<nombre de la colección>, opciones)

Donde el primer parámetro es el nombre que le queremos asignar a la colección, y los siguientes, todos opcionales, pueden ser (entre otros):

Campo	Tipo	Descripción
`capped`	Booleano	Si es `true` , permite una colección limitada. Una colección limitada es una colección de tamaño fijo que sobrescribe automáticamente sus entradas más antiguas cuando alcanza su tamaño máximo. Si especifica `true` , también debe especificar el parámetro de `size` .
`autoIndexId`	Booleano	Si es `true` crea automáticamente un índice en el campo `_id` . Por defecto es `false`
`size`	Número	Especifica el tamaño máximo en bytes para una colección limitada. Es obligatorio si el campo `capped` está a `true` .
`max`	Número	Especifica el número máximo de documentos que están permitidos en la colección limitada.

> use test
switched to db test

> db.createCollection("mycollection")
{ "ok" : 1 }

> db.createCollection("mycol", {capped : true, autoIndexId: true, size: 6142800, max: 10000})
{
    "note" : "the autoIndexId option is deprecated and will be removed in a future release",
    "ok" : 1
}

> show collections
mycol
mycollection

Como se ha visto anteriormente al crear la base de datos, podemos insertar un documento en una colección sin que la hayamos creado anteriormente. Esto es porque MongoDB crea automáticamente una colección cuando insertas algún documento en ella:

> db.tutorialspoint.insert({"name":"tutorialspoint"})
WriteResult({ "nInserted" : 1 })

> show collections
mycol
mycollection
tutorialspoint

¶Borrar colección

Para borrar una colección basta con situarnos en la base de datos que la contiene, y ejecutar lo siguiente:

db.<nombre_de_la_colección>.drop()

> db.mycollection.drop()
true

> show collections
mycol
tutorialspoint

¶Insertar documento

Para insertar datos en una colección de MongoDB necesitaremos usar el método insert() o save().

Ejemplo del método insert:

> db.colection.insert({
... title: 'Esto es una prueba para MDAD',
... description: 'MongoDB es una BD no SQL',
... by: 'Classmate and Me',
... tags: ['mongodb', 'database'],
... likes: 100
... })
WriteResults({ "nInserted" : 1 })

En este ejemplo solo se ha insertado un único documento, pero podemos insertar los que queramos separándolos de la siguiente forma:

db.collection.insert({documento}, {documento2}, {documento3})

No hace falta especificar un ID ya que el propio mongo asigna un ID a cada documento automáticamente, aunque nos da la opción de poder asignarle uno mediante el atributo _id en la inserción de los datos

Como se indica en el título de este apartado también se puede insertar mediante el método db.coleccion.save(documento), funcionando este como el método insert.

¶Método `find()`

El método find en MongoDB es el que nos permite realizar consultas a las colecciones de nuestra base de datos:

db.<nombre_de_la_colección>.find()

Este método mostrará de una forma no estructurada todos los documentos de la colección. Si le añadimos la función pretty a este método, se mostrarán de una manera más «bonita».

> db.colection.find()
{ "_id": ObjectId("5e738f0989f85a7eafdf044a"), "title" : "Esto es una prueba para MDAD", "description" : "MongoDB es una BD no SQL", "by" : "Classmate and Me", "tags" : [ "mongodb", "database" ], "likes" : 100 }

> db.colection.find().pretty()
{
    "_id": ObjectId("5e738f0989f85a7eafdf044a"),
    "title" : "Esto es una prueba para MDAD",
    "description" : "MongoDB es una BD no SQL",
    "by" : "Classmate and Me",
    "tags" : [
        "mongodb",
        "database"
    ],
    "likes" : 100
}

Los equivalentes del where en las bases de datos relacionales son:

Operación	Sintaxis	Ejemplo	Equivalente en RDBMS
Igual	`{<clave>:<valor>}`	`db.mycol.find({"by":"Classmate and Me"})`	`where by = 'Classmate and Me'`
Menor que	`{<clave>:{$lt:<valor>}}`	`db.mycol.find({"likes":{$lt:60}})`	`where likes < 60`
Menor o igual que	`{<clave>:{$lte:<valor>}}`	`db.mycol.find({"likes":{$lte:60}})`	`where likes <= 60`
Mayor que	`{<clave>:{$gt:<valor>}}`	`db.mycol.find({"likes":{$gt:60}})`	`where likes > 60`
Mayor o igual que	`{<clave>:{$gte:<valor>}}`	`db.mycol.find({"likes":{$gte:60}})`	`where likes >= 60`
No igual	`{<clave>:{$ne:<valor>}}`	`db.mycol.find({"likes":{$ne:60}})`	`where likes != 60`

En el método find() podemos añadir condiciones AND y OR de la siguiente manera:

(AND)
> db.colection.find({$and:[{"by":"Classmate and Me"},{"title": "Esto es una prueba para MDAD"}]}).pretty()

(OR)
> db.colection.find({$or:[{"by":"Classmate and Me"},{"title": "Esto es una prueba para MDAD"}]}).pretty()

(Ambos a la vez)
> db.colection.find({"likes": {$gt:10}, $or: [{"by": "Classmate and Me"}, {"title": "Esto es una prueba para MDAD"}]}).pretty()

La última llamada con ambos a la vez equivalente en una consulta SQL a:

where likes>10 AND (by = 'Classmate and Me' OR title = 'Esto es una prueba para MDAD')

¶Actualizar un documento

En MongoDB se hace utilizando el método update:

db.<nombre_colección>.update(<criterio_de_selección>, <dato_actualizado>)

Para este ejemplo vamos a actualizar el documento que hemos insertado en el apartado anterior:

> db.colection.update({'title':'Esto es una prueba para MDAD'},{$set:{'title':'Título actualizado'}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.colection.find().pretty()
{
    "_id": ObjectId("5e738f0989f85a7eafdf044a"),
    "title" : "Título actualizado",
    "description" : "MongoDB es una BD no SQL",
    "by" : "Classmate and Me",
    "tags" : [
        "mongodb",
        "database"
    ],
    "likes" : 100
}

Anteriormente se ha mencionado el método save() para la inserción de documentos, pero también podemos utilizarlo para sustituir documentos enteros por uno nuevo:

> db.<nombre_de_la_colección>.save({_id:ObjectId(), <nuevo_documento>})

Con nuestro documento:

> db.colection.save(
...   {
...     "_id": ObjectId("5e738f0989f85a7eafdf044a"), "title": "Este es el nuevo título", "by": "MDAD"
...   }
... )
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

> db.colection.find()
{
    "_id": ObjectId("5e738f0989f85a7eafdf044a"),
    "title": "Este es el nuevo título",
    "by": "MDAD"
}

¶Borrar documento

Para borrar un documento utilizaremos el método remove() de la siguiente manera:

db.<nombre_de_la_colección>.remove(<criterio_de_borrado>)

Considerando la colección del apartado anterior borraremos el único documento que tenemos:

> db.colection.remove({'title': 'Este es el nuevo título'})
WriteResult({ "nRemoved" : 1 })
> db.colection.find().pretty()
>

Para borrar todos los documentos de una colección usamos:

db.<colección>.remove({})

¶Indexación

MongDB nos permite crear índices sobre atributos de una colección de la siguiente forma:

db.<colección>.createIndex( {<atributo>:<opciones>})

Como ejemplo:

> db.mycol.createIndex({"title":1})
{
    "createdCollectionAutomatically" : false,
    "numIndexesBefore" : 1,
    "numIndexesAfter" : 2,
    "ok" : 1
}

Si queremos más de un atributo en el índice lo haremos así:

> db.mycol.ensureIndex({"title":1,"description":-1})

Los valores que puede tomar son +1 para ascendente o -1 para descendente.

¶Referencias

Manual MongoDB. (n.d.). https://docs.mongodb.com/manual/
MongoDB Tutorial – Tutorialspoint. (n.d.). – https://www.tutorialspoint.com/mongodb/index.htm

MongoDB: Introducción

2020-03-19T23:00:00+00:00

MongoDB: Introducción

Este es el primer post en la serie sobre Mongo, en el cuál introduciremos dicha bases de datos NoSQL y veremos sus características e instalación.

Created 2020-03-05
Modified 2020-03-20

Otros posts en esta serie:

MongoDB: Introducción (este post)
MongoDB: Operaciones Básicas y Arquitectura

Este post está hecho en colaboración con un compañero.

¶Definición

MongoDB es una base de datos orientada a documentos. Esto quiere decir que en lugar de guardar los datos en registros, guarda los datos en documentos. Estos documentos son almacenados en BSON, que es una representación binaria de JSON. Una de las principales diferencias respecto a las bases de datos relacionales es que no necesita seguir ningún esquema, los documentos de una misma colección pueden tener esquemas diferentes.

MongoDB está escrito en C++, aunque las consultas se hacen pasando objetos JSON como parámetro.

{
        "_id" : ObjectId("52f602d787945c344bb4bda5"),
        "name" : "Tyrion",
        "hobbies" : [ 
            "books", 
            "girls", 
            "wine"
        ],
        "friends" : [ 
            {
                "name" : "Bronn",
                "ocuppation" : "sellsword"
            }, 
            {
                "name" : "Shae",
                "ocuppation" : "handmaiden"
            }
        ]
 }

¶Características

MongoDB alcanza un balance perfecto entre rendimiento y funcionalidad gracias a su sistema de consulta de contenidos. Pero sus características principales no se limitan solo a esto, también cuenta con otras que lo posicionan como el preferido de muchos desarrolladores de aplicaciones como aplicaciones móviles, gaming, logging o e-commerce.

Algunas de las principales características de esta base de datos son:

Almacenamiento orientado a documentos (documentos JSON con esquemas dinámicos).
Soporte Full index: puede crear índices sobre cualquier atributo y añadir múltiples índices secundarios.
Replicación y alta disponibilidad: espejos entre LANs y WANs.
Auto-Sharding: escalabilidad horizontal sin comprometer la funcionalidad, está limitada, actualmente, a 20 nodos, aunque el objetivo es alcanzar una cifra cercana a los 1000.
Consultas ricas y basadas en documentos.
Rápidas actualizaciones en el contexto.
Soporte comercial, capacitación y consultoría disponibles.
También puede ser utilizada para el almacenamiento de archivos aprovechando la capacidad de MongoDB para el balanceo de carga y la replicación de datos.

En cuanto a la arquitectura, podríamos decir que divide en tres partes: las bases de datos, las colecciones y los documentos (que contienen los campos de cada entrada).

Base de datos: cada una de las bases de datos tiene un conjunto propio de archivos en el sistema de archivos con diversas bases de datos existentes en un solo servidor.
Colección: un conjunto de documentos de base de datos. El equivalente RDBMS de la colección es una tabla. Toda colección existe dentro de una única base de datos.
Documento: un conjunto de pares clave/valor. Los documentos están asociados con esquemas dinámicos. La ventaja de tener esquemas dinámicos es que el documento en una sola colección no tiene que tener la misma estructura o campos.

¶Arista dentro del Teorema CAP

MongoDB es CP por defecto, es decir, garantiza consistencia y tolerancia a particiones (fallos). Pero también podemos configurar el nivel de consistencia, eligiendo el número de nodos a los que se replicarán los datos. O podemos configurar si se pueden leer datos de los nodos secundarios (en MongoDB solo hay un servidor principal, que es el único que acepta inserciones o modificaciones). Si permitimos leer de un nodo secundario mediante la replicación, sacrificamos consistencia, pero ganamos disponibilidad.

¶Descarga e instalación

¶Windows

Descargar el archivo desde https://www.mongodb.com/download-center#production

Doble clic en el archivo .msi
El instalador de Windows lo guía a través del proceso de instalación. Si elige la opción de instalación personalizada, puede especificar un directorio de instalación. MongoDB no tiene ninguna otra dependencia del sistema. Puede instalar y ejecutar MongoDB desde cualquier carpeta que elija.
Ejecutar el .exe que hemos instalado.

¶Linux

Abrimos una terminal y ejecutamos:

sudo apt-get update
sudo apt install -y mongodb-org

Luego comprobamos el estado del servicio:

sudo systemctl start mongod
sudo systemctl status mongod

Finalmente ejecutamos la base de datos con el comando:

sudo mongo

¶macOS

Abrimos una terminal y ejecutamos:

brew update
brew install mongodb

Iniciamos el servicio:

brew services start mongodb

¶Referencias

Cassandra: Operaciones Básicas y Arquitectura

2020-03-19T23:00:00+00:00

Cassandra: Operaciones Básicas y Arquitectura

Este es el segundo post en la serie sobre Cassandra, con una breve descripción de las operaciones básicas (tales como inserción, recuperación e indexado), y ejecución por completo junto con el modelo de datos y arquitectura.

Created 2020-03-05
Modified 2020-03-20

Otros posts en esta serie:

Cassandra: Introducción
Cassandra: Operaciones Básicas y Arquitectura (este post)

Este post está hecho en colaboración con un compañero.

Antes de poder ejecutar ninguna consulta, debemos lanzar la base de datos en caso de que no se encuentre en ejecución aún. Para ello, en una terminal, lanzamos el binario de cassandra:

$ cassandra-3.11.6/bin/cassandra

Sin cerrar esta consola, abrimos otra en la que podamos usar la CQL shell:

$ cassandra-3.11.6/bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.6 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh>

¶Crear

¶Crear una base de datos

Cassandra denomina a las «bases de datos» como «espacio de claves» (keyspace en inglés).

cqlsh> create keyspace helloworld with replication = {'class': 'SimpleStrategy', 'replication_factor': 3};

Cuando creamos un nuevo keyspace, indicamos el nombre y la estrategia de replicación a usar. Nosotros usamos la estrategia simple con un factor 3 de replicación.

¶Crear una tabla

Una vez estemos dentro de un keyspace, podemos crear tablas. Vamos a crear una tabla llamada «greetings» con identificador (número entero), mensaje (texto) y lenguaje (varchar).

cqlsh> use helloworld;
cqlsh:helloworld> create table greetings(id int primary key, message text, lang varchar);

¶Crear una fila

Insertar nuevas filas es similar a otros sistemas gestores de datos, mediante la sentencia INSERT:

cqlsh:helloworld> insert into greetings(id, message, lang) values(1, '¡Bienvenido!', 'es');
cqlsh:helloworld> insert into greetings(id, message, lang) values(2, 'Welcome!', 'es');

¶Leer

La lectura se lleva a cabo mediante la sentencia SELECT:

cqlsh:helloworld> select * from greetings;

 id | lang | message
----+------+--------------
  1 |   es | ¡Bienvenido!
  2 |   es |     Welcome!

(2 rows)

cqlsh colorea la salida, lo cuál resulta muy útil para identificar la clave primaria y distintos tipos de datos como texto, cadenas o números:

¶Actualizar

La actualización se lleva a cabo con la sentencia UPDATE. Vamos a arreglar el fallo que hemos cometido al insertar «Welcome!» como español:

cqlsh:helloworld> update greetings set lang = 'en' where id = 2;

¶Indexar

cqlsh:helloworld> create index langIndex on greetings(lang);

¶Borrar

Finalmente, el borrado se lleva a cabo con la sentencia DELETE. Es posible borrar solo campos individuales, lo cuál los pone a nulos:

cqlsh:helloworld> delete message from greetings where id = 1;

Para eliminar la fila entera, basta con no especificar la columna:

cqlsh:helloworld> delete from greetings where id = 1;

¶Referencias

Visualizing Cáceres’ OpenData

2020-03-18T23:00:00+00:00

Visualizing Cáceres’ OpenData

The city of Cáceres has online services to provide Open Data over a wide range of categories, all of which are very interesting to explore!

Created 2020-03-09
Modified 2020-03-19

We have chosen two different datasets, and will explore four different ways to visualize the data.

This post is co-authored with Classmate.

¶Obtain the data

We are interested in the JSON format for the census in 2017 and those for the vias of the city. This way, we can explore the population and their location in interesting ways! You may follow those two links and select the JSON format under Resources to download it.

Why JSON? We will be using Python (3.7 or above) and matplotlib for quick iteration, and loading the data with Python’s json module will be trivial.

¶Implementation

¶Imports and constants

We are going to need a lot of things in this code, such as json to load the data, matplotlib to visualize it, and other data types and type hinting for use in the code.

We also want automatic download of the JSON files if they’re missing, so we add their URLs and download paths as constants.

import json
import re
import os
import sys
import urllib.request
import matplotlib.pyplot as plt
from dataclasses import dataclass
from collections import namedtuple
from datetime import date
from pathlib import Path
from typing import Optional

CENSUS_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionCENSUS&year=2017&format=json'
VIAS_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:Via&format=json'

CENSUS_JSON = Path('data/demografia/Padrón_Cáceres_2017.json')
VIAS_JSON = Path('data/via/Vías_Cáceres.json')

¶Data classes

Parse, don’t validate. By defining a clear data model, we will be able to tell at a glance what information we have available. It will also be typed, so we won’t be confused as to what is what! Python 3.7 introduces [dataclasses](https://docs.python.org/3/library/dataclasses.html), which are a wonderful feature to define… well, data classes concisely.

We also have a [namedtuple](https://docs.python.org/3/library/collections.html#collections.namedtuple) for points, because it’s extremely common to represent them as tuples.

Point = namedtuple('Point', 'long lat')

@dataclass
class Census:
    year: int
    via: int
    count_per_year: dict
    count_per_city: dict
    count_per_gender: dict
    count_per_nationality: dict
    time_year: int

@dataclass
class Via:
    name: str
    kind: str
    code: int
    history: Optional[str]
    old_name: Optional[str]
    length: Optional[float]
    start: Optional[Point]
    middle: Optional[Point]
    end: Optional[Point]
    geometry: Optional[list]

¶Helper methods

We will have a little helper method to automatically download the JSON when missing. This is just for convenience, we could as well just download it manually. But it is fun to automate things.

def ensure_file(file, url):
    if not file.is_file():
        print('Downloading', file.name, 'because it was missing...', end='', flush=True, file=sys.stderr)
        file.parent.mkdir(parents=True, exist_ok=True)
        urllib.request.urlretrieve(url, file)
        print(' Done.', file=sys.stderr)

¶Parsing the data

I will be honest, parsing Cáceres’ OpenData is a pain in the neck! The official descriptions are huge and not all that helpful. Maybe if one needs documentation for a specific field. But luckily for us, the names are pretty self-descriptive, and we can explore the data to get a feel for what we will find.

We define two methods, one to iterate over Census values, and another to iterate over Via values. Here’s where our friend [re](https://docs.python.org/3/library/re.html) comes in, and oh boy the format of the data…

For example, the year and via identifier are best extracted from the URI! The information is also available in the rdfs_label field, but that’s just a Spanish text! At least the URI will be more reliable… hopefully.

Birth date. They could have used a JSON list, but nah, that would’ve been too simple. Instead, you are given a string separated by semicolons. The values? They could have been dictionaries with names for «year» and «age», but nah! That would’ve been too simple! Instead, you are given strings that look like «2001 (7)», and that’s the year and the count.

The birth place? Sometimes it’s «City (Province) (Count)», but sometimes the province is missing. Gender? Semicolon-separated. And there are only two genders. I know a few people who would be upset just reading this, but it’s not my data, it’s theirs. Oh, and plenty of things are optional. That was a lot of AttributeError: 'NoneType' object has no attribute 'foo' to work through!

But as a reward, we have nicely typed data, and we no longer have to deal with this mess when trying to visualize it. For brevity, we will only be showing how to parse the census data, and not the data for the vias. This post is already long enough on its own.

def iter_census(file):
    with file.open() as fd:
        data = json.load(fd)

    for row in data['results']['bindings']:
        year, via = map(int, row['uri']['value'].split('/')[-1].split('-'))

        count_per_year = {}
        for item in row['schema_birthDate']['value'].split(';'):
            y, c = map(int, re.match(r'(\d+) \((\d+)\)', item).groups())
            count_per_year[y] = c

        count_per_city = {}
        for item in row['schema_birthPlace']['value'].split(';'):
            match = re.match(r'([^(]+) \(([^)]+)\) \((\d+)\)', item)
            if match:
                l, _province, c = match.groups()
            else:
                l, c = re.match(r'([^(]+) \((\d+)\)', item).groups()

            count_per_city[l] = int(c)

        count_per_gender = {}
        for item in row['foaf_gender']['value'].split(';'):
            g, c = re.match(r'([^(]+) \((\d+)\)', item).groups()
            count_per_gender[g] = int(c)

        count_per_nationality = {}
        for item in row['schema_nationality']['value'].split(';'):
            match = re.match(r'([^(]+) \((\d+)\)', item)
            if match:
                g, c = match.groups()
            else:
                g, _alt_name, c = re.match(r'([^(]+) \(([^)]+)\) \((\d+)\)', item).groups()

            count_per_nationality[g] = int(c)
        time_year = int(row['time_year']['value'])

        yield Census(
            year=year,
            via=via,
            count_per_year=count_per_year,
            count_per_city=count_per_city,
            count_per_gender=count_per_gender,
            count_per_nationality=count_per_nationality,
            time_year=time_year,
        )

¶Visualizing the data

Here comes the fun part! After parsing all the desired data from the mentioned JSON files, we plotted the data in four different graphics making use of Python’s matplotlib library. This powerful library helps with the creation of different visualizations in Python.

¶Visualizing the genders in a pie chart

After seeing that there are only two genders in the data of the census, we, displeased, started work in a chart for it. The pie chart was the best option since we wanted to show only the percentages of each gender. The result looks like this:

Pretty straight forward, isn’t it? To display this wonderful graphic, we used the following code:

def pie_chart(ax, data):
    lists = sorted(data.items())

    x, y = zip(*lists)
    ax.pie(y, labels=x, autopct='%1.1f%%',
            shadow=True, startangle=90)
    ax.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.

We pass the axis as the input parameter (later we will explain why) and the data collected from the JSON regarding the genders, which are in a dictionary with the key being the labels and the values the tally of each gender. We sort the data and with some unpacking magic we split it into two values: x being the labels and y the amount of each gender.

After that we plot the pie chart with the data and labels from y and x, we specify that we want the percentage with one decimal place with the autopct parameter, we enable shadows for the presentation, and specify the start angle at 90º.

¶Date tick labels

We wanted to know how many of the living people were born on each year, so we are making a date plot! In the census we have the year each person was born in, and using that information is an easy task after parsing the data (parsing was an important task of this work). The result looks as follows:

How did we do this? The following code was used:

def date_tick(ax, data):
    lists = sorted(data.items())

    x, y = zip(*lists)
    x = [date(year, 1, 1) for year in x]
    ax.plot(x, y)

Again, we pass in an axis and the data related with the year born, we sort it, split it into two lists, being the keys the years and the values the number per year. After that, we put the years in a date format for the plot to be more accurate. Finally, we plot the values into that wonderful graphic.

¶Stacked bar chart

We wanted to know if there was any relation between the latitudes and count per gender, so we developed the following code:

def stacked_bar_chart(ax, data):
    labels = []
    males = []
    females = []

    for latitude, genders in data.items():
        labels.append(str(latitude))
        males.append(genders['Male'])
        females.append(genders['Female'])

    ax.bar(labels, males, label='Males')
    ax.bar(labels, females, bottom=males, label='Females')

    ax.set_ylabel('Counts')
    ax.set_xlabel('Latitudes')
    ax.legend()

The key of the data dictionary is the latitude rounded to two decimals, and value is another dictionary, which is composed by the key that is the name of the gender and the value, the number of people per gender. So, in a single entry of the data dictionary we have the latitude and how many people per gender are in that latitude.

We iterate the dictionary to extract the different latitudes and people per gender (because we know only two genders are used, we hardcode it to two lists). Then we plot them putting the males and females lists at the bottom and set the labels of each axis. The result is the following:

¶Scatter plots

This last graphic was very tricky to get right. It’s incredibly hard to find the extent of a city online! We were getting confused because some of the points were way farther than the centre of Cáceres, and the city background is a bit stretched even if the coordinates appear correct. But in the end, we did a pretty good job on it.

def scatter_map(ax, data):
    xs = []
    ys = []
    areas = []
    for (long, lat), count in data.items():
        xs.append(long)
        ys.append(lat)
        areas.append(count / 100)

    if CACERES_MAP.is_file():
        ax.imshow(plt.imread(str(CACERES_MAP)), extent=CACERES_EXTENT)
    else:
        print('Note:', CACERES_MAP, 'does not exist, not showing it', file=sys.stderr)

    ax.scatter(xs, ys, areas, alpha=0.1)

This time, the keys in the data dictionary are points and the values are the total count of people in that point. We use a normal for loop to create the different lists. For the areas on how big the circles we are going to represent will be, we divide the count of people by some number, like 100, or otherwise they would be huge.

If the file of the map is present, we render it so that we can get a sense on where the points are, but if the file is missing we print a warning.

At last, we draw the scatter plot with some low alpha value (there’s a lot of overlapping points). The result is absolutely gorgeous. (For some definitions of gorgeous, anyway):

Just for fun, here’s what it looks like if we don’t divide the count by 100 and lower the opacity to 0.01:

That’s a big solid blob, and the opacity is only set to 0.01!

¶Drawing all the graphs in the same window

To draw all the graphs in the same window instead of getting four different windows we made use of the subplots function, like this:

fig, axes = plt.subplots(2, 2)

This will create a matrix of two by two of axes that we store in the axes variable (fitting name!). Following this code are the different calls to the methods commented before, where we access each individual axis and pass it to the methods to draw on:

pie_chart(axes[0, 0], genders)
date_tick(axes[0, 1], years)
stacked_bar_chart(axes[1, 0], latitudes)
scatter_map(axes[1, 1], positions)

Lastly, we plot the different graphics:

plt.show()

Wrapping everything together, here’s the result:

The numbers in some of the graphs are a bit crammed together, but we’ll blame that on matplotlib.

¶Closing words

Wow, that was a long journey! We hope that this post helped you pick some interest in data exploration, it’s such a fun world. We also offer the full download for the code below, because we know it’s quite a bit!

Which of the graphs was your favourite? I personally like the count per date, I think it’s nice to see the growth. Let us know in the comments below!

download removed

What is an algorithm?

2020-03-17T23:00:00+00:00

What is an algorithm?

Algorithms are a sequence of instructions that can be followed to achieve something. That something can be anything, and depends entirely on your problem!

Created 2020-02-25
Modified 2020-03-18

For example, a recipe to cook some really nice food is an algorithm: it guides you, step by step, to cook something nice. People dealing with mathemathics also apply algorithms to transform their data. And computers love algorithms, too!

In reality, any computer program can basically be thought as an algorithm. It contains a series of instructions for the computer to execute. Running them is a process that takes time, consumes input and produces output. This is also why terms like «procedure» come up when talking about them.

Computer programs (their algorithms) are normally written in some more specific language, like Java or Python. The instructions are very clear here, which is what we need! A natural language like English is a lot harder to process, and ambiguous. I’m sure you’ve been in arguments because the other person didn’t understand you!

¶References

algorithm – definition and meaning: https://www.wordnik.com/words/algorithm
Algorithm: https://en.wikipedia.org/wiki/Algorithm
What is a «computer algorithm»?: https://computer.howstuffworks.com/what-is-a-computer-algorithm.htm

Introduction to NoSQL

2020-03-17T23:00:00+00:00

Introduction to NoSQL

This post will primarly focus on the talk held in the GOTO 2012 conference: Introduction to NoSQL by Martin Fowler. It can be seen as an informal, summarized transcript of the talk

Created 2020-02-25
Modified 2020-03-18

The relational database model is affected by the impedance mismatch problem. This occurs because we have to match our high-level design with the separate columns and rows used by relational databases.

Taking the in-memory objects and putting them into a relational database (which were dominant at the time) simply didn’t work out. Why? Relational databases were more than just databases, they served as a an integration mechanism across applications, up to the 2000s. For 20 years!

With the rise of the Internet and the sheer amount of traffic, databases needed to scale. Unfortunately, relational databases only scale well vertically (by upgrading a single node). This is very expensive, and not something many could afford.

The problem are those pesky JOIN‘s, and its friends GROUP BY. Because our program and reality model don’t match the tables used by SQL, we have to rely on them to query the data. It is because the model doesn’t map directly.

Furthermore, graphs don’t map very well at all to relational models.

We needed a way to scale horizontally (by increasing the amount of nodes), something relational databases were not designed to do.

We need to do something different, relational across nodes is an unnatural act

This inspired the NoSQL movement.

#nosql was only meant to be a hashtag to advertise it, but unfortunately it’s how it is called now

It is not possible to define NoSQL, but we can identify some of its characteristics:

Non-relational
Cluster-friendly (this was the original spark)
Open-source (until now, generally)
21st century web culture
Schema-less (easier integration or conjugation of several models, structure aggregation)

These databases use different data models to those used by the relational model. However, it is possible to identify 4 broad chunks (some may say 3, or even 2!):

Key-value store. With a certain key, you obtain the value corresponding to it. It knows nothing else, nor does it care. We say the data is opaque.
Document-based. It stores an entire mass of documents with complex structure, normally through the use of JSON (XML has been left behind). Then, you can ask for certain fields, structures, or portions. We say the data is transparent.
Column-family. There is a «row key», and within it we store multiple «column families» (columns that fit together, our aggregate). We access by row-key and column-family name.

All of these kind of serve to store documents without any explicit schema. Just shove in anything! This gives a lot of flexibility and ease of migration, except… that’s not really true. There’s an implicit schema when querying.

For example, a query where we may do anOrder['price'] * anOrder['quantity'] is assuming that anOrder has both a price and a quantity, and that both of these can be multiplied together. «Schema-less» is a fuzzy term.

However, it is the lack of a fixed schema that gives flexibility.

One could argue that the line between key-value and document-based is very fuzzy, and they would be right! Key-value databases often let you include additional metadata that behaves like an index, and in document-based, documents often have an identifier anyway.

The common notion between these three types is what matters. They save an entire structure as an unit. We can refer to these as «Aggregate Oriented Databases». Aggregate, because we group things when designing or modeling our systems, as opposed to relational databases that scatter the information across many tables.

There exists a notable outlier, though, and that’s:

Graph databases. They use a node-and-arc graph structure. They are great for moving on relationships across things. Ironically, relational databases are not very good at jumping across relationships! It is possibly to perform very interesting queries in graph databases which would be really hard and costly on relational models. Unlike the aggregated databases, graphs break things into even smaller units. NoSQL is not the solution. It depends on how you’ll work with your data. Do you need an aggregate database? Will you have a lot of relationships? Or would the relational model be good fit for you?

NoSQL, however, is a good fit for large-scale projects (data will always grow) and faster development (the impedance mismatch is drastically reduced).

Regardless of our choice, it is important to remember that NoSQL is a young technology, which is still evolving really fast (SQL has been stable for decades). But the polyglot persistence is what matters. One must know the alternatives, and be able to choose.

Relational databases have the well-known ACID properties: Atomicity, Consistency, Isolation and Durability.

NoSQL (except graph-based!) are about being BASE instead: Basically Available, Soft state, Eventual consistency.

SQL needs transactions because we don’t want to perform a read while we’re only half-way done with a write! The readers and writers are the problem, and ensuring consistency results in a performance hit, even if the risk is low (two writers are extremely rare but it still must be handled).

NoSQL on the other hand doesn’t need ACID because the aggregate is the transaction boundary. Even before NoSQL itself existed! Any update is atomic by nature. When updating many documents it is a problem, but this is very rare.

We have to distinguish between logical and replication consistency. During an update and if a conflict occurs, it must be resolved to preserve the logical consistency. Replication consistency on the other hand is preserveed when distributing the data across many machines, for example during sharding or copies.

Replication buys us more processing power and resillence (at the cost of more storage) in case some of the nodes die. But what happens if what dies is the communication across the nodes? We could drop the requests and preserve the consistency, or accept the risk to continue and instead preserve the availability.

The choice on whether trading consistency for availability is acceptable or not depends on the domain rules. It is the domain’s choice, the business people will choose. If you’re Amazon, you always want to be able to sell, but if you’re a bank, you probably don’t want your clients to have negative numbers in their account!

Regardless of what we do, in a distributed system, the CAP theorem always applies: Consistecy, Availability, Partitioning-tolerancy (error tolerancy). It is impossible to guarantee all 3 at 100%. Most of the times, it does work, but it is mathematically impossible to guarantee at 100%.

A database has to choose what to give up at some point. When designing a distributed system, this must be considered. Normally, the choice is made between consistency or response time.

¶Further reading

Big Data

2020-03-17T23:00:00+00:00

Big Data

Big Data sounds like a buzzword you may be hearing everywhere, but it’s actually here to stay!

Created 2020-02-25
Modified 2020-03-18

¶What is Big Data?

And why is it so important? We use this term to refer to the large amount of data available, rapidly growing every day, that cannot be processed in conventional ways. It’s not only about the amount, it’s also about the variety and rate of growth.

Thanks to technological advancements, there are new ways to process this insane amount of data, which would otherwise be too costly for processing in traditional database systems.

¶Where does data come from?

It can be pictures in your phone, industry transactions, messages in social networks, a sensor in the mountains. It can come from anywhere, which makes the data very varied.

Just to give some numbers, over 12TB of data is generated on Twitter daily. If you purchase a laptop today (as of March 2020), the disk will be roughly 1TB, maybe 2TB. Twitter would fill 6 of those drives every day!

What about Facebook? It is estimated they store around 100PB of photos and videos. That would be 50000 laptop disks. Not a small number. And let’s not talk about worldwide network traffic…

¶What data can be exploited?

So, we have a lot of data. Should we attempt and process everything? We can distinguish several categories.

Web and Social Media: Clickstream Data, Twitter Feeds, Facebook Postings, Web content… Stuff coming from social networks.
Biometrics: Facial Recognion, Genetics… Any kind of personal recognition.
Machine-to-Machine: Utility Smart Meter Readings, RFID Readings, Oil Rig Sensor Readings, GPS Signals… Any sensor shared with other machines.
Human Generated: Call Center Voice Recordings, Email, Electronic Medical Records… Even the voice notes one sends over WhatsApp count.
Big Transaction Data: Healthcare Claims, Telecommunications Call Detail Records, Utility Billing Records… Financial transactions.

But asking what to process is asking the wrong question. Instead, one should think about «What problem am I trying to solve?».

¶How to exploit this data?

What are some of the ways to deal with this data? If the problem fits the Map-Reduce paradigm then Hadoop is a great option! Hadoop is inspired by Google File System (GFS), and achieves great parallelism across the nodes of a cluster, and has the following components:

Hadoop Distributed File System. Data is divided into smaller «blocks» and distributed across the cluster, which makes it possible to execute the mapping and reduction in smaller subsets, and makes it possible to scale horizontally.
Hadoop MapReduce. First, a data set is «mapped» into a different set, and data becomes a list of tuples (key, value). The «reduce» step works on these tuples and combines them into a smaller subset.
Hadoop Common. These are a set of libraries that ease working with Hadoop.

¶Key insights

Big Data is a field whose goal is to extract information from very large sets of data, and find ways to do so. To summarize its different dimensions, we can refer to what’s known as «the Four V’s of Big Data»:

Volume. Really large quantities.
Velocity. Processing response time matters!
Variety. Data comes from plenty of sources.
Veracity. Can we trust all sources, though?

Some sources talk about a fifth V for Value; because processing this data is costly, it is important we can get value out of it.

…And some other sources go as high as seven V’s, including Viability and Visualization. Computers can’t take decissions on their own (yet), a human has to. And they can only do so if they’re presented the data (and visualize it) in a meaningful way.

¶Infographics

Let’s see some pictures, we all love pictures:

¶Common patterns

¶References

¿Qué es Big Data? – https://www.ibm.com/developerworks/ssa/local/im/que-es-big-data/
The Four V’s of Big Data – https://www.ibmbigdatahub.com/infographic/four-vs-big-data
Big data – https://en.wikipedia.org/wiki/Big_data
Las 5 V’s del Big Data – https://www.quanticsolutions.es/big-data/las-5-vs-del-big-data
Las 7 V del Big data: Características más importantes – https://www.iic.uam.es/innovacion/big-data-caracteristicas-mas-importantes-7-v/

pagong

Data Mining and Data Warehousing

¶Data Mining and Data Warehousing

Privado: Final NoSQL evaluation

¶Classmate’s Evaluation

A practical example with Hadoop

¶Installation

¶Processing data

Developing a Python application for Cassandra

¶Our goal

¶Data model

¶Dependencies

¶The code

¶Getting users

¶Saving users

¶Fetching users

¶Parsing arguments

¶All together

¶References

Introduction to Hadoop and its MapReduce

¶Milestones

¶Why not use RDBMS?

¶Major components

¶MapReduce

¶References

Data Warehousing and OLAP

¶Types of data

¶Places to store data

¶Ways to store data

¶Key takeaways

¶References

Cassandra: Introducción

¶Finalidad de la tecnología

¶Cómo funciona

¶Características

¶Arista dentro del Teorema CAP

¶Descarga

¶Instalación

¶Referencias

Privado: NoSQL evaluation

¶Classmate’s Evaluation

¶Classmate’s Evaluation

Mining of Massive Datasets

MongoDB: Operaciones Básicas y Arquitectura

¶Creación de una base de datos e inserción de un primer documento

¶Borrar base de datos

¶Crear colección

¶Borrar colección

¶Insertar documento

¶Método find()

¶Actualizar un documento

¶Borrar documento

¶Indexación

¶Referencias

MongoDB: Introducción

¶Definición

¶Características

¶Arista dentro del Teorema CAP

¶Descarga e instalación

¶Windows

¶Linux

¶macOS

¶Referencias

Cassandra: Operaciones Básicas y Arquitectura

¶Crear

¶Crear una base de datos

¶Crear una tabla

¶Crear una fila

¶Leer

¶Actualizar

¶Indexar

¶Borrar

¶Referencias

Visualizing Cáceres’ OpenData

¶Obtain the data

¶Implementation

¶Imports and constants

¶Data classes

¶Helper methods

¶Parsing the data

¶Método `find()`