1<feed xmlns="http://www.w3.org/2005/Atom"><title>pagong</title><id>pagong</id><updated>2020-10-02T22:00:00+00:00</updated><entry><title>Information Retrieval and Web Search</title><id>dist/index/index.html</id><updated>2020-10-02T22:00:00+00:00</updated><published>2020-10-02T22:00:00+00:00</published><summary>During 2020 at university, this subject ("Recuperación de la Información y Búsqueda en la Web")</summary><content type="html" src="dist/index/index.html">&lt;!DOCTYPE html&gt;
   2&lt;html&gt;
   3&lt;head&gt;
   4&lt;meta charset=&quot;utf-8&quot; /&gt;
   5&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
   6&lt;title&gt;Information Retrieval and Web Search&lt;/title&gt;
   7&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
   8&lt;/head&gt;
   9&lt;body&gt;
  10&lt;main&gt;
  11&lt;h1 class=&quot;title&quot; id=&quot;information_retrieval_and_web_search&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#information_retrieval_and_web_search&quot;&gt;¶&lt;/a&gt;Information Retrieval and Web Search&lt;/h1&gt;
  12&lt;div class=&quot;date-created-modified&quot;&gt;2020-10-03&lt;/div&gt;
  13&lt;p&gt;During 2020 at university, this subject (&amp;quot;Recuperación de la Información y Búsqueda en la Web&amp;quot;)
  14had us write blog posts as assignments. I think it would be really fun and I wanted to preserve
  15that work here, with the hopes it's interesting to someone.&lt;/p&gt;
  16&lt;p&gt;The posts were auto-generated from the original HTML files and manually anonymized later.&lt;/p&gt;
  17&lt;/main&gt;
  18&lt;/body&gt;
  19&lt;/html&gt;
  20 </content></entry><entry><title>Privado: Final NoSQL evaluation</title><id>dist/final-nosql-evaluation/index.html</id><updated>2020-05-13T22:00:00+00:00</updated><published>2020-05-12T22:00:00+00:00</published><summary>This evaluation is a bit different to my </summary><content type="html" src="dist/final-nosql-evaluation/index.html">&lt;!DOCTYPE html&gt;
  21&lt;html&gt;
  22&lt;head&gt;
  23&lt;meta charset=&quot;utf-8&quot; /&gt;
  24&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
  25&lt;title&gt;Privado: Final NoSQL evaluation&lt;/title&gt;
  26&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
  27&lt;/head&gt;
  28&lt;body&gt;
  29&lt;main&gt;
  30&lt;p&gt;This evaluation is a bit different to my &lt;a href=&quot;/blog/ribw/16/nosql-evaluation/&quot;&gt;previous one&lt;/a&gt; because this time I have been tasked to evaluate the student &lt;code&gt;a(i - 2)&lt;/code&gt;, and because I am &lt;code&gt;a = 9&lt;/code&gt; that happens to be &lt;code&gt;a(7) =&lt;/code&gt; Classmate.&lt;/p&gt;
  31&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-05-13&lt;br&gt;
  32Modified 2020-05-14&lt;/div&gt;
  33&lt;p&gt;Unfortunately for Classmate, the only entry related to NoSQL I have found in their blog is Prima y segunda Actividad: Base de datos NoSQL which does not develop an application as requested for the third entry (as of 14th of May).&lt;/p&gt;
  34&lt;p&gt;This means that, instead, I will evaluate &lt;code&gt;a(i - 3)&lt;/code&gt; which happens to be &lt;code&gt;a(6) =&lt;/code&gt; Classmate and they do have an entry.&lt;/p&gt;
  35&lt;h2 class=&quot;title&quot; id=&quot;classmate_s_evaluation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#classmate_s_evaluation&quot;&gt;¶&lt;/a&gt;Classmate’s Evaluation&lt;/h2&gt;
  36&lt;p&gt;&lt;strong&gt;Grading: B.&lt;/strong&gt;&lt;/p&gt;
  37&lt;p&gt;The post I have evaluated is BB.DD. NoSQL RethinkDB 3ª Fase. Aplicación.&lt;/p&gt;
  38&lt;p&gt;It starts with an introduction, properly explaining what database they have chosen and why, but not what application they will be making.&lt;/p&gt;
  39&lt;p&gt;This is detailed just below in the next section, although it’s a bit vague.&lt;/p&gt;
  40&lt;p&gt;The next section talks about the Python dependencies that are required, but they never said they would be making a Python application or that we need to install Python!&lt;/p&gt;
  41&lt;p&gt;The next section talks about the file structure of the project, and they detail what everything part does, although I have missed some code snippets.&lt;/p&gt;
  42&lt;p&gt;The final result is pretty cool and contains many interesting graphs, they provide a download to the source code and list all the relevant references used.&lt;/p&gt;
  43&lt;p&gt;Except for a weird «necesario falta» in the text, it’s otherwise well-written, although given the issues above I cannot grade it with the highest score.&lt;/p&gt;
  44&lt;/main&gt;
  45&lt;/body&gt;
  46&lt;/html&gt;
  47 </content></entry><entry><title>Developing a Python application for MongoDB</title><id>dist/developing-a-python-application-for-mongodb/index.html</id><updated>2020-04-15T22:00:00+00:00</updated><published>2020-03-24T23:00:00+00:00</published><summary>This is the third and last post in the MongoDB series, where we will develop a Python application to process and store OpenData inside Mongo.</summary><content type="html" src="dist/developing-a-python-application-for-mongodb/index.html">&lt;!DOCTYPE html&gt;
  48&lt;html&gt;
  49&lt;head&gt;
  50&lt;meta charset=&quot;utf-8&quot; /&gt;
  51&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
  52&lt;title&gt;Developing a Python application for MongoDB&lt;/title&gt;
  53&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
  54&lt;/head&gt;
  55&lt;body&gt;
  56&lt;main&gt;
  57&lt;p&gt;This is the third and last post in the MongoDB series, where we will develop a Python application to process and store OpenData inside Mongo.&lt;/p&gt;
  58&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-25&lt;br&gt;
  59Modified 2020-04-16&lt;/div&gt;
  60&lt;p&gt;Other posts in this series:&lt;/p&gt;
  61&lt;ul&gt;
  62&lt;li&gt;&lt;a href=&quot;/blog/ribw/mongodb-an-introduction/&quot;&gt;MongoDB: an Introduction&lt;/a&gt;&lt;/li&gt;
  63&lt;li&gt;&lt;a href=&quot;/blog/ribw/mongodb-basic-operations-and-architecture/&quot;&gt;MongoDB: Basic Operations and Architecture&lt;/a&gt;&lt;/li&gt;
  64&lt;li&gt;&lt;a href=&quot;/blog/ribw/developing-a-python-application-for-mongodb/&quot;&gt;Developing a Python application for MongoDB&lt;/a&gt; (this post)&lt;/li&gt;
  65&lt;/ul&gt;
  66&lt;p&gt;This post is co-authored wih a Classmate.&lt;/p&gt;
  67&lt;hr /&gt;
  68&lt;h2 class=&quot;title&quot; id=&quot;what_are_we_making_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#what_are_we_making_&quot;&gt;¶&lt;/a&gt;What are we making?&lt;/h2&gt;
  69&lt;p&gt;We are going to develop a web application that renders a map, in this case, the town of Cáceres, with which users can interact. When the user clicks somewhere on the map, the selected location will be sent to the server to process. This server will perform geospatial queries to Mongo and once the results are ready, the information is presented back at the webpage.&lt;/p&gt;
  70&lt;p&gt;The data used for the application comes from &lt;a href=&quot;https://opendata.caceres.es/&quot;&gt;Cáceres’ OpenData&lt;/a&gt;, and our goal is that users will be able to find information about certain areas in a quick and intuitive way, such as precise coordinates, noise level, and such.&lt;/p&gt;
  71&lt;h2 id=&quot;what_are_we_using_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#what_are_we_using_&quot;&gt;¶&lt;/a&gt;What are we using?&lt;/h2&gt;
  72&lt;p&gt;The web application will be using &lt;a href=&quot;https://python.org/&quot;&gt;Python&lt;/a&gt; for the backend, &lt;a href=&quot;https://svelte.dev/&quot;&gt;Svelte&lt;/a&gt; for the frontend, and &lt;a href=&quot;https://www.mongodb.com/&quot;&gt;Mongo&lt;/a&gt; as our storage database and processing center.&lt;/p&gt;
  73&lt;ul&gt;
  74&lt;li&gt;&lt;strong&gt;Why Python?&lt;/strong&gt; It’s a comfortable language to write and to read, and has a great ecosystem with &lt;a href=&quot;https://pypi.org/&quot;&gt;plenty of libraries&lt;/a&gt;.&lt;/li&gt;
  75&lt;li&gt;&lt;strong&gt;Why Svelte?&lt;/strong&gt; Svelte is the New Thing&lt;strong&gt;™&lt;/strong&gt; in the world of component frameworks for JavaScript. It is similar to React or Vue, but compiled and with a lot less boilerplate. Check out their &lt;a href=&quot;https://svelte.dev/blog/svelte-3-rethinking-reactivity&quot;&gt;Svelte post&lt;/a&gt; to learn more.&lt;/li&gt;
  76&lt;li&gt;&lt;strong&gt;Why Mongo?&lt;/strong&gt; We believe NoSQL is the right approach for doing the kind of processing and storage that we expect, and it’s &lt;a href=&quot;https://docs.mongodb.com/&quot;&gt;very easy to use&lt;/a&gt;. In addition, we will be making Geospatial Queries which &lt;a href=&quot;https://docs.mongodb.com/manual/geospatial-queries/&quot;&gt;Mongo supports&lt;/a&gt;.&lt;/li&gt;
  77&lt;/ul&gt;
  78&lt;p&gt;Why didn’t we choose to make a smaller project, you may ask? You will be shocked to hear that we do not have an answer for that!&lt;/p&gt;
  79&lt;p&gt;Note that we will not be embedding &lt;strong&gt;all&lt;/strong&gt; the code of the project in this post, or it would be too long! We will include only the relevant snippets needed to understand the core ideas of the project, and not the unnecessary parts of it (for example, parsing configuration files to easily change the port where the server runs is not included).&lt;/p&gt;
  80&lt;h2 id=&quot;python_dependencies&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#python_dependencies&quot;&gt;¶&lt;/a&gt;Python dependencies&lt;/h2&gt;
  81&lt;p&gt;Because we will program it in Python, you need Python installed. You can install it using a package manager of your choice or heading over to the &lt;a href=&quot;https://www.python.org/downloads/&quot;&gt;Python downloads section&lt;/a&gt;, but if you’re on Linux, chances are you have it installed already.&lt;/p&gt;
  82&lt;p&gt;Once Python 3.7 or above is installed, install &lt;a href=&quot;https://motor.readthedocs.io/en/stable/&quot;&gt;&lt;code&gt;motor&lt;/code&gt; (Asynchronous Python driver for MongoDB)&lt;/a&gt; and the &lt;a href=&quot;https://docs.aiohttp.org/en/stable/web.html&quot;&gt;&lt;code&gt;aiohttp&lt;/code&gt; server&lt;/a&gt; through &lt;code&gt;pip&lt;/code&gt;:&lt;/p&gt;
  83&lt;pre&gt;&lt;code&gt;pip install aiohttp motor
  84&lt;/code&gt;&lt;/pre&gt;
  85&lt;p&gt;Make sure that Mongo is running in the background (this has been described in previous posts), and we should be able to get to work.&lt;/p&gt;
  86&lt;h2 id=&quot;web_dependencies&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#web_dependencies&quot;&gt;¶&lt;/a&gt;Web dependencies&lt;/h2&gt;
  87&lt;p&gt;To work with Svelte and its dependencies, we will need &lt;code&gt;[npm](https://www.npmjs.com/)&lt;/code&gt; which comes with &lt;a href=&quot;https://nodejs.org/en/&quot;&gt;NodeJS&lt;/a&gt;, so go and &lt;a href=&quot;https://nodejs.org/en/download/&quot;&gt;install Node from their site&lt;/a&gt;. The download will be different depending on your operating system.&lt;/p&gt;
  88&lt;p&gt;Following &lt;a href=&quot;https://svelte.dev/blog/the-easiest-way-to-get-started&quot;&gt;the easiest way to get started with Svelte&lt;/a&gt;, we will put our project in a &lt;code&gt;client/&lt;/code&gt; folder (because this is what the clients see, the frontend). Feel free to tinker a bit with the configuration files to change the name and such, although this isn’t relevant for the rest of the post.&lt;/p&gt;
  89&lt;h2 id=&quot;finding_the_data&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#finding_the_data&quot;&gt;¶&lt;/a&gt;Finding the data&lt;/h2&gt;
  90&lt;p&gt;We are going to work with the JSON files provided by &lt;a href=&quot;http://opendata.caceres.es/&quot;&gt;OpenData Cáceres&lt;/a&gt;. In particular, we want information about the noise, census, vias and trees. To save you the time from &lt;a href=&quot;http://opendata.caceres.es/dataset&quot;&gt;searching each of these&lt;/a&gt;, we will automate the download with code.&lt;/p&gt;
  91&lt;p&gt;If you want to save the data offline or just know what data we’ll be using for other purposes though, you can right click on the following links and select «Save Link As…» with the name of the link:&lt;/p&gt;
  92&lt;ul&gt;
  93&lt;li&gt;&lt;code&gt;[noise.json](http://opendata.caceres.es/GetData/GetData?dataset=om:MedicionRuido&amp;amp;format=json)&lt;/code&gt;&lt;/li&gt;
  94&lt;li&gt;&lt;code&gt;[census.json](http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionPadron&amp;amp;year=2017&amp;amp;format=json)&lt;/code&gt;&lt;/li&gt;
  95&lt;li&gt;&lt;code&gt;[vias.json](http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionPadron&amp;amp;year=2017&amp;amp;format=json)&lt;/code&gt;&lt;/li&gt;
  96&lt;li&gt;&lt;code&gt;[trees.json](http://opendata.caceres.es/GetData/GetData?dataset=om:Arbol&amp;amp;format=json)&lt;/code&gt;&lt;/li&gt;
  97&lt;/ul&gt;
  98&lt;h2 id=&quot;backend&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#backend&quot;&gt;¶&lt;/a&gt;Backend&lt;/h2&gt;
  99&lt;p&gt;It’s time to get started with some code! We will put it in a &lt;code&gt;server/&lt;/code&gt; folder because it will contain the Python server, that is, the backend of our application.&lt;/p&gt;
 100&lt;p&gt;We are using &lt;code&gt;aiohttp&lt;/code&gt; because we would like our server to be &lt;code&gt;async&lt;/code&gt;. We don’t expect a lot of users at the same time, but it’s good to know our server would be well-designed for that use-case. As a bonus, it makes IO points clear in the code, which can help reason about it. The implicit synchronization between &lt;code&gt;await&lt;/code&gt; is also a nice bonus.&lt;/p&gt;
 101&lt;h3 id=&quot;saving_the_data_in_mongo&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#saving_the_data_in_mongo&quot;&gt;¶&lt;/a&gt;Saving the data in Mongo&lt;/h3&gt;
 102&lt;p&gt;Before running the server, we must ensure that the data we need is already stored and indexed in Mongo. Our &lt;code&gt;server/data.py&lt;/code&gt; will take care of downloading the files, cleaning them up a little (Cáceres’ OpenData can be a bit awkward sometimes), inserting them into Mongo and indexing them.&lt;/p&gt;
 103&lt;p&gt;Downloading the JSON data can be done with &lt;code&gt;[ClientSession.get](https://aiohttp.readthedocs.io/en/stable/client_reference.html#aiohttp.ClientSession.get)&lt;/code&gt;. We also take this opportunity to clean up the messy encoding from the JSON, which does not seem to be UTF-8 in some cases.&lt;/p&gt;
 104&lt;pre&gt;&lt;code&gt;async def load_json(session, url):
 105    fixes = [(old, new.encode('utf-8')) for old, new in [
 106        (b'\xc3\x83\\u2018', 'Ñ'),
 107        (b'\xc3\x83\\u0081', 'Á'),
 108        (b'\xc3\x83\\u2030', 'É'),
 109        (b'\xc3\x83\\u008D', 'Í'),
 110        (b'\xc3\x83\\u201C', 'Ó'),
 111        (b'\xc3\x83\xc5\xa1', 'Ú'),
 112        (b'\xc3\x83\xc2\xa1', 'á'),
 113    ]]
 114
 115    async with session.get(url) as resp:
 116        data = await resp.read()
 117
 118    # Yes, this feels inefficient, but it's not really worth improving.
 119    for old, new in fixes:
 120        data = data.replace(old, new)
 121
 122    data = data.decode('utf-8')
 123    return json.loads(data)
 124&lt;/code&gt;&lt;/pre&gt;
 125&lt;p&gt;Later on, it can be reused for the various different URLs:&lt;/p&gt;
 126&lt;pre&gt;&lt;code&gt;import aiohttp
 127
 128NOISE_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:MedicionRuido&amp;amp;format=json'
 129# (...other needed URLs here)
 130
 131async def insert_to_db(db):
 132    async with aiohttp.ClientSession() as session:
 133        data = await load_json(session, NOISE_URL)
 134        # now we have the JSON data cleaned up, ready to be parsed
 135&lt;/code&gt;&lt;/pre&gt;
 136&lt;h3 id=&quot;data_model&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#data_model&quot;&gt;¶&lt;/a&gt;Data model&lt;/h3&gt;
 137&lt;p&gt;With the JSON data in our hands, it’s time to parse it. Always remember to &lt;a href=&quot;https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/&quot;&gt;parse, don’t validate&lt;/a&gt;. With &lt;a href=&quot;https://docs.python.org/3/library/dataclasses.html&quot;&gt;Python 3.7 &lt;code&gt;dataclasses&lt;/code&gt;&lt;/a&gt; it’s trivial to define classes that will store only the fields we care about, typed, and with proper names:&lt;/p&gt;
 138&lt;pre&gt;&lt;code&gt;from dataclasses import dataclass
 139
 140Longitude = float
 141Latitude = float
 142
 143@dataclass
 144class GSON:
 145    type: str
 146    coordinates: (Longitude, Latitude)
 147
 148@dataclass
 149class Noise:
 150    id: int
 151    geo: GSON
 152    level: float
 153&lt;/code&gt;&lt;/pre&gt;
 154&lt;p&gt;This makes it really easy to see that, if we have a &lt;code&gt;Noise&lt;/code&gt;, we can access its &lt;code&gt;geo&lt;/code&gt; data which is a &lt;code&gt;GSON&lt;/code&gt; with a &lt;code&gt;type&lt;/code&gt; and &lt;code&gt;coordinates&lt;/code&gt;, having &lt;code&gt;Longitude&lt;/code&gt; and &lt;code&gt;Latitude&lt;/code&gt; respectively. &lt;code&gt;dataclasses&lt;/code&gt; and &lt;code&gt;[typing](https://docs.python.org/3/library/typing.html)&lt;/code&gt; make dealing with this very easy and clear.&lt;/p&gt;
 155&lt;p&gt;Every dataclass will be on its own collection inside Mongo, and these are:&lt;/p&gt;
 156&lt;ul&gt;
 157&lt;li&gt;
 158&lt;p&gt;Noise&lt;/p&gt;
 159&lt;/li&gt;
 160&lt;li&gt;
 161&lt;p&gt;Integer &lt;code&gt;id&lt;/code&gt;&lt;/p&gt;
 162&lt;/li&gt;
 163&lt;li&gt;
 164&lt;p&gt;GeoJSON &lt;code&gt;geo&lt;/code&gt;&lt;/p&gt;
 165&lt;/li&gt;
 166&lt;li&gt;
 167&lt;p&gt;String &lt;code&gt;type&lt;/code&gt;&lt;/p&gt;
 168&lt;/li&gt;
 169&lt;li&gt;
 170&lt;p&gt;Longitude-latitude pair &lt;code&gt;coordinates&lt;/code&gt;&lt;/p&gt;
 171&lt;/li&gt;
 172&lt;li&gt;
 173&lt;p&gt;Floating-point number &lt;code&gt;level&lt;/code&gt;&lt;/p&gt;
 174&lt;/li&gt;
 175&lt;li&gt;
 176&lt;p&gt;Tree&lt;/p&gt;
 177&lt;/li&gt;
 178&lt;li&gt;
 179&lt;p&gt;String &lt;code&gt;name&lt;/code&gt;&lt;/p&gt;
 180&lt;/li&gt;
 181&lt;li&gt;
 182&lt;p&gt;String &lt;code&gt;gender&lt;/code&gt;&lt;/p&gt;
 183&lt;/li&gt;
 184&lt;li&gt;
 185&lt;p&gt;Integer &lt;code&gt;units&lt;/code&gt;&lt;/p&gt;
 186&lt;/li&gt;
 187&lt;li&gt;
 188&lt;p&gt;Floating-point number &lt;code&gt;height&lt;/code&gt;&lt;/p&gt;
 189&lt;/li&gt;
 190&lt;li&gt;
 191&lt;p&gt;Floating-point number &lt;code&gt;cup_diameter&lt;/code&gt;&lt;/p&gt;
 192&lt;/li&gt;
 193&lt;li&gt;
 194&lt;p&gt;Floating-point number &lt;code&gt;trunk_diameter&lt;/code&gt;&lt;/p&gt;
 195&lt;/li&gt;
 196&lt;li&gt;
 197&lt;p&gt;Optional string &lt;code&gt;variety&lt;/code&gt;&lt;/p&gt;
 198&lt;/li&gt;
 199&lt;li&gt;
 200&lt;p&gt;Optional string &lt;code&gt;distribution&lt;/code&gt;&lt;/p&gt;
 201&lt;/li&gt;
 202&lt;li&gt;
 203&lt;p&gt;GeoJSON &lt;code&gt;geo&lt;/code&gt;&lt;/p&gt;
 204&lt;/li&gt;
 205&lt;li&gt;
 206&lt;p&gt;Optional string &lt;code&gt;irrigation&lt;/code&gt;&lt;/p&gt;
 207&lt;/li&gt;
 208&lt;li&gt;
 209&lt;p&gt;Census&lt;/p&gt;
 210&lt;/li&gt;
 211&lt;li&gt;
 212&lt;p&gt;Integer &lt;code&gt;year&lt;/code&gt;&lt;/p&gt;
 213&lt;/li&gt;
 214&lt;li&gt;
 215&lt;p&gt;Via &lt;code&gt;via&lt;/code&gt;&lt;/p&gt;
 216&lt;/li&gt;
 217&lt;li&gt;
 218&lt;p&gt;String &lt;code&gt;name&lt;/code&gt;&lt;/p&gt;
 219&lt;/li&gt;
 220&lt;li&gt;
 221&lt;p&gt;String &lt;code&gt;kind&lt;/code&gt;&lt;/p&gt;
 222&lt;/li&gt;
 223&lt;li&gt;
 224&lt;p&gt;Integer &lt;code&gt;code&lt;/code&gt;&lt;/p&gt;
 225&lt;/li&gt;
 226&lt;li&gt;
 227&lt;p&gt;Optional string &lt;code&gt;history&lt;/code&gt;&lt;/p&gt;
 228&lt;/li&gt;
 229&lt;li&gt;
 230&lt;p&gt;Optional string &lt;code&gt;old_name&lt;/code&gt;&lt;/p&gt;
 231&lt;/li&gt;
 232&lt;li&gt;
 233&lt;p&gt;Optional floating-point number &lt;code&gt;length&lt;/code&gt;&lt;/p&gt;
 234&lt;/li&gt;
 235&lt;li&gt;
 236&lt;p&gt;Optional GeoJSON &lt;code&gt;start&lt;/code&gt;&lt;/p&gt;
 237&lt;/li&gt;
 238&lt;li&gt;
 239&lt;p&gt;GeoJSON &lt;code&gt;middle&lt;/code&gt;&lt;/p&gt;
 240&lt;/li&gt;
 241&lt;li&gt;
 242&lt;p&gt;Optional GeoJSON &lt;code&gt;end&lt;/code&gt;&lt;/p&gt;
 243&lt;/li&gt;
 244&lt;li&gt;
 245&lt;p&gt;Optional list with geometry pairs &lt;code&gt;geometry&lt;/code&gt;&lt;/p&gt;
 246&lt;/li&gt;
 247&lt;li&gt;
 248&lt;p&gt;Integer &lt;code&gt;count&lt;/code&gt;&lt;/p&gt;
 249&lt;/li&gt;
 250&lt;li&gt;
 251&lt;p&gt;Mapping year-to-count &lt;code&gt;count_per_year&lt;/code&gt;&lt;/p&gt;
 252&lt;/li&gt;
 253&lt;li&gt;
 254&lt;p&gt;Mapping gender-to-count &lt;code&gt;count_per_gender&lt;/code&gt;&lt;/p&gt;
 255&lt;/li&gt;
 256&lt;li&gt;
 257&lt;p&gt;Mapping nationality-to-count &lt;code&gt;count_per_nationality&lt;/code&gt;&lt;/p&gt;
 258&lt;/li&gt;
 259&lt;li&gt;
 260&lt;p&gt;Integer &lt;code&gt;time_year&lt;/code&gt;&lt;/p&gt;
 261&lt;/li&gt;
 262&lt;/ul&gt;
 263&lt;p&gt;Now, let’s define a method to actually parse the JSON and yield instances from these new data classes:&lt;/p&gt;
 264&lt;pre&gt;&lt;code&gt;@classmethod
 265def iter_from_json(cls, data):
 266    for row in data['results']['bindings']:
 267        noise_id = int(row['uri']['value'].split('/')[-1])
 268        long = float(row['geo_long']['value'])
 269        lat = float(row['geo_lat']['value'])
 270        level = float(row['om_nivelRuido']['value'])
 271
 272        yield cls(
 273            id=noise_id,
 274            geo=GSON(type='Point', coordinates=[long, lat]),
 275            level=level
 276        )
 277&lt;/code&gt;&lt;/pre&gt;
 278&lt;p&gt;Here we iterate over the input JSON &lt;code&gt;data&lt;/code&gt; bindings and &lt;code&gt;yield cls&lt;/code&gt; instances with more consistent naming than the original one. We also extract the data from the many unnecessary nested levels of the JSON and have something a lot flatter to work with.&lt;/p&gt;
 279&lt;p&gt;For those of you who don’t know what &lt;code&gt;yield&lt;/code&gt; does (after all, not everyone is used to seeing generators), here’s two functions that work nearly the same:&lt;/p&gt;
 280&lt;pre&gt;&lt;code&gt;def squares_return(n):
 281    result = []
 282    for i in range(n):
 283        result.append(n ** 2)
 284    return result
 285
 286def squares_yield(n):
 287    for i in range(n):
 288        yield n ** 2
 289&lt;/code&gt;&lt;/pre&gt;
 290&lt;p&gt;The difference is that the one with &lt;code&gt;yield&lt;/code&gt; is «lazy» and doesn’t need to do all the work up-front. It will generate (yield) more values as they are needed when you use a &lt;code&gt;for&lt;/code&gt; loop. Generally, it’s a better idea to create generator functions than do all the work early which may be unnecessary. See &lt;a href=&quot;https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do&quot;&gt;What does the «yield» keyword do?&lt;/a&gt; if you still have questions.&lt;/p&gt;
 291&lt;p&gt;With everything parsed, it’s time to insert the data into Mongo. If the data was not present yet (0 documents), then we will download the file, parse it, insert it as documents into the given Mongo &lt;code&gt;db&lt;/code&gt;, and index it:&lt;/p&gt;
 292&lt;pre&gt;&lt;code&gt;from dataclasses import asdict
 293
 294async def insert_to_db(db):
 295    async with aiohttp.ClientSession() as session:
 296        if await db.noise.estimated_document_count() == 0:
 297            data = await load_json(session, NOISE_URL)
 298
 299            await db.noise.insert_many(asdict(noise) for noise in Noise.iter_from_json(data))
 300            await db.noise.create_index([('geo', '2dsphere')])
 301&lt;/code&gt;&lt;/pre&gt;
 302&lt;p&gt;We repeat this process for all the other data, and just like that, Mongo is ready to be used in our server.&lt;/p&gt;
 303&lt;h3 id=&quot;indices&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#indices&quot;&gt;¶&lt;/a&gt;Indices&lt;/h3&gt;
 304&lt;p&gt;In order to execute our geospatial queries we have to create an index on the attribute that represents the location, because the operators that we will use requires it. This attribute can be a &lt;a href=&quot;https://docs.mongodb.com/manual/reference/geojson/&quot;&gt;GeoJSON object&lt;/a&gt; or a legacy coordinate pair.&lt;/p&gt;
 305&lt;p&gt;We have decided to use a GeoJSON object because we want to avoid legacy features that may be deprecated in the future.&lt;/p&gt;
 306&lt;p&gt;The attribute is called &lt;code&gt;geo&lt;/code&gt; for the &lt;code&gt;Tree&lt;/code&gt; and &lt;code&gt;Noise&lt;/code&gt; objects and &lt;code&gt;start&lt;/code&gt;, &lt;code&gt;middle&lt;/code&gt; or &lt;code&gt;end&lt;/code&gt; for the &lt;code&gt;Via&lt;/code&gt; class. In the &lt;code&gt;Via&lt;/code&gt; we are going to index the attribute &lt;code&gt;middle&lt;/code&gt; because it is the most representative field for us. Because the &lt;code&gt;Via&lt;/code&gt; is inside the &lt;code&gt;Census&lt;/code&gt; and it doesn’t have its own collection, we create the index on the &lt;code&gt;Census&lt;/code&gt; collection.&lt;/p&gt;
 307&lt;p&gt;The used index type is &lt;code&gt;2dsphere&lt;/code&gt; because it supports queries that work on geometries on an earth-like sphere. Another option is the &lt;code&gt;2d&lt;/code&gt; index but it’s not a good fit for our because it is for queries that calculate geometries on a two-dimensional plane.&lt;/p&gt;
 308&lt;h3 id=&quot;running_the_server&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#running_the_server&quot;&gt;¶&lt;/a&gt;Running the server&lt;/h3&gt;
 309&lt;p&gt;If we ignore the configuration part of the server creation, our &lt;code&gt;server.py&lt;/code&gt; file is pretty simple. Its job is to create a &lt;a href=&quot;https://aiohttp.readthedocs.io/en/stable/web.html&quot;&gt;server application&lt;/a&gt;, setup Mongo and return it to the caller so that they can run it:&lt;/p&gt;
 310&lt;pre&gt;&lt;code&gt;import asyncio
 311import subprocess
 312import motor.motor_asyncio
 313
 314from aiohttp import web
 315
 316from . import rest, data
 317
 318def create_app():
 319    ret = subprocess.run('npm run build', cwd='../client', shell=True).returncode
 320    if ret != 0:
 321        exit(ret)
 322
 323    db = motor.motor_asyncio.AsyncIOMotorClient().opendata
 324    loop = asyncio.get_event_loop()
 325    loop.run_until_complete(data.insert_to_db(db))
 326
 327    app = web.Application()
 328    app['db'] = db
 329
 330    app.router.add_routes([
 331        web.get('/', lambda r: web.HTTPSeeOther('/index.html')),
 332        *rest.ROUTES,
 333        web.static('/', os.path.join(config['www']['root'], 'public')),
 334    ])
 335
 336    return app
 337&lt;/code&gt;&lt;/pre&gt;
 338&lt;p&gt;There’s a bit going on here, but it’s nothing too complex:&lt;/p&gt;
 339&lt;ul&gt;
 340&lt;li&gt;We automatically run &lt;code&gt;npm run build&lt;/code&gt; on the frontend because it’s very comfortable to have the frontend built automatically before the server runs.&lt;/li&gt;
 341&lt;li&gt;We create a Motor client and access the &lt;code&gt;opendata&lt;/code&gt; database. Into it, we load the data, effectively saving it in Mongo for the server to use.&lt;/li&gt;
 342&lt;li&gt;We create the server application and save a reference to the Mongo database in it, so that it can be used later on any endpoint without needing to recreate it.&lt;/li&gt;
 343&lt;li&gt;We define the routes of our app: root, REST and static (where the frontend files live). We’ll get to the &lt;code&gt;rest&lt;/code&gt; part soon.
 344Running the server is now simple:&lt;/li&gt;
 345&lt;/ul&gt;
 346&lt;pre&gt;&lt;code&gt;def main():
 347    from aiohttp import web
 348    from . import server
 349
 350    app = server.create_app()
 351    web.run_app(app)
 352
 353if __name__ == '__main__':
 354    main()
 355&lt;/code&gt;&lt;/pre&gt;
 356&lt;h3 id=&quot;rest_endpoints&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#rest_endpoints&quot;&gt;¶&lt;/a&gt;REST endpoints&lt;/h3&gt;
 357&lt;p&gt;The frontend will communicate with the backend via &lt;a href=&quot;https://en.wikipedia.org/wiki/Representational_state_transfer&quot;&gt;REST&lt;/a&gt; calls, so that it can ask for things like «give me the information associated with this area», and the web server can query the Mongo server to reply with a HTTP response. This little diagram should help:&lt;/p&gt;
 358&lt;p&gt;&lt;img src=&quot;bitmap.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
 359&lt;p&gt;What we need to do, then, is define those REST endpoints we mentioned earlier when creating the server. We will process the HTTP request, ask Mongo for the data, and return the HTTP response:&lt;/p&gt;
 360&lt;pre&gt;&lt;code&gt;import asyncio
 361import pymongo
 362
 363from aiohttp import web
 364
 365async def get_area_info(request):
 366    try:
 367        long = float(request.query['long'])
 368        lat = float(request.query['lat'])
 369        distance = float(request.query['distance'])
 370    except KeyError as e:
 371        raise web.HTTPBadRequest(reason=f'a required parameter was missing: {e.args[0]}')
 372    except ValueError:
 373        raise web.HTTPBadRequest(reason='one of the parameters was not a valid float')
 374
 375    geo_avg_noise_pipeline = [{
 376        '$geoNear': {
 377            'near' : {'type': 'Point', 'coordinates': [long, lat]},
 378            'maxDistance': distance,
 379            'minDistance': 0,
 380            'spherical' : 'true',
 381            'distanceField' : 'distance'
 382        }
 383    }]
 384
 385    db = request.app['db']
 386
 387    try:
 388        noise_count, sum_noise, avg_noise = 0, 0, 0
 389        async for item in db.noise.aggregate(geo_avg_noise_pipeline):
 390            noise_count += 1
 391            sum_noise += item['level']
 392
 393        if noise_count != 0:
 394            avg_noise = sum_noise / noise_count
 395        else:
 396            avg_noise = None
 397
 398    except pymongo.errors.ConnectionFailure:
 399        raise web.HTTPServiceUnavailable(reason='no connection to database')
 400
 401    return web.json_response({
 402        'tree_count': tree_count,
 403        'trees_per_type': [[k, v] for k, v in trees_per_type.items()],
 404        'census_count': census_count,
 405        'avg_noise': avg_noise,
 406    })
 407
 408ROUTES = [
 409    web.get('/rest/get-area-info', get_area_info)
 410]
 411&lt;/code&gt;&lt;/pre&gt;
 412&lt;p&gt;In this code, we’re only showing how to return the average noise because that’s the simplest we can do. The real code also fetches tree count, tree count per type, and census count.&lt;/p&gt;
 413&lt;p&gt;Again, there’s quite a bit to go through, so let’s go step by step:&lt;/p&gt;
 414&lt;ul&gt;
 415&lt;li&gt;We parse the frontend’s &lt;code&gt;request.query&lt;/code&gt; into &lt;code&gt;float&lt;/code&gt; that we can use. In particular, the frontend is asking us for information at a certain latitude, longitude, and distance. If the query is malformed, we return a proper error.&lt;/li&gt;
 416&lt;li&gt;We create our query for Mongo outside, just so it’s clearer to read.&lt;/li&gt;
 417&lt;li&gt;We access the database reference we stored earlier when creating the server with &lt;code&gt;request.app['db']&lt;/code&gt;. Handy!&lt;/li&gt;
 418&lt;li&gt;We try to query Mongo. It may fail if the Mongo server is not running, so we should handle that and tell the client what’s happening. If it succeeds though, we will gather information about the average noise.&lt;/li&gt;
 419&lt;li&gt;We return a &lt;code&gt;json_response&lt;/code&gt; with Mongo results for the frontend to present to the user.
 420You may have noticed we defined a &lt;code&gt;ROUTES&lt;/code&gt; list at the bottom. This will make it easier to expand in the future, and the server creation won’t need to change anything in its code, because it’s already unpacking all the routes we define here.&lt;/li&gt;
 421&lt;/ul&gt;
 422&lt;h3 id=&quot;geospatial_queries&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#geospatial_queries&quot;&gt;¶&lt;/a&gt;Geospatial queries&lt;/h3&gt;
 423&lt;p&gt;In order to retrieve the information from Mongo database we have defined two geospatial queries:&lt;/p&gt;
 424&lt;pre&gt;&lt;code&gt;geo_query = {
 425    '$nearSphere' : {
 426        '$geometry': {
 427            'type': 'Point',
 428            'coordinates': [long, lat]
 429         },
 430        '$maxDistance': distance,
 431        '$minDistance': 0
 432    }
 433}
 434&lt;/code&gt;&lt;/pre&gt;
 435&lt;p&gt;This query uses &lt;a href=&quot;https://docs.mongodb.com/manual/reference/operator/query/nearSphere/#op._S_nearSphere&quot;&gt;the operator &lt;code&gt;$nearSphere&lt;/code&gt;&lt;/a&gt; which return geospatial objects in proximity to a point on a sphere.&lt;/p&gt;
 436&lt;p&gt;The sphere point is represented by the &lt;code&gt;$geometry&lt;/code&gt; operator where it is specified the type of geometry and the coordinates (given by the HTTP request).&lt;/p&gt;
 437&lt;p&gt;The maximum and minimum distance are represented by &lt;code&gt;$maxDistance&lt;/code&gt; and &lt;code&gt;$minDistance&lt;/code&gt; respectively. We specify that the maximum distance is the radio selected by the user.&lt;/p&gt;
 438&lt;pre&gt;&lt;code&gt;geo_avg_noise_pipeline = [{
 439    '$geoNear': {
 440        'near' : {'type': 'Point', 'coordinates': [long, lat]},
 441        'maxDistance': distance,
 442        'minDistance': 0,
 443        'spherical' : 'true',
 444        'distanceField' : 'distance'
 445    }
 446}]
 447&lt;/code&gt;&lt;/pre&gt;
 448&lt;p&gt;This query uses the &lt;a href=&quot;https://docs.mongodb.com/manual/core/aggregation-pipeline/&quot;&gt;aggregation pipeline&lt;/a&gt; stage &lt;a href=&quot;https://docs.mongodb.com/manual/reference/operator/aggregation/geoNear/#pipe._S_geoNear&quot;&gt;&lt;code&gt;$geoNear&lt;/code&gt;&lt;/a&gt; which returns an ordered stream of documents based on the proximity to a geospatial point. The output documents include an additional distance field.&lt;/p&gt;
 449&lt;p&gt;The &lt;code&gt;near&lt;/code&gt; field is mandatory and is the point for which to find the closest documents. In this field it is specified the type of geometry and the coordinates (given by the HTTP request).&lt;/p&gt;
 450&lt;p&gt;The &lt;code&gt;distanceField&lt;/code&gt; field is also mandatory and is the output field that will contain the calculated distance. In this case we’ve just called it &lt;code&gt;distance&lt;/code&gt;.&lt;/p&gt;
 451&lt;p&gt;Some other fields are &lt;code&gt;maxDistance&lt;/code&gt; that indicates the maximum allowed distance from the center of the point, &lt;code&gt;minDistance&lt;/code&gt; for the minimum distance, and &lt;code&gt;spherical&lt;/code&gt; which tells MongoDB how to calculate the distance between two points.&lt;/p&gt;
 452&lt;p&gt;We specify the maximum distance as the radio selected by the user in the frontend.&lt;/p&gt;
 453&lt;h2 id=&quot;frontend&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#frontend&quot;&gt;¶&lt;/a&gt;Frontend&lt;/h2&gt;
 454&lt;p&gt;As said earlier, our frontend will use Svelte. We already downloaded the template, so we can start developing. For some, this is the most fun part, because they can finally see and interact with some of the results. But for this interaction to work, we needed a functional backend which we now have!&lt;/p&gt;
 455&lt;h3 id=&quot;rest_queries&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#rest_queries&quot;&gt;¶&lt;/a&gt;REST queries&lt;/h3&gt;
 456&lt;p&gt;The frontend has to query the server to get any meaningful data to show on the page. The &lt;a href=&quot;https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API&quot;&gt;Fetch API&lt;/a&gt; does not throw an exception if the server doesn’t respond with HTTP OK, but we would like one if things go wrong, so that we can handle them gracefully. The first we’ll do is define our own exception &lt;a href=&quot;https://stackoverflow.com/a/27724419&quot;&gt;which is not pretty&lt;/a&gt;:&lt;/p&gt;
 457&lt;pre&gt;&lt;code&gt;function NetworkError(message, status) {
 458    var instance = new Error(message);
 459    instance.name = 'NetworkError';
 460    instance.status = status;
 461    Object.setPrototypeOf(instance, Object.getPrototypeOf(this));
 462    if (Error.captureStackTrace) {
 463        Error.captureStackTrace(instance, NetworkError);
 464    }
 465    return instance;
 466}
 467
 468NetworkError.prototype = Object.create(Error.prototype, {
 469    constructor: {
 470        value: Error,
 471        enumerable: false,
 472        writable: true,
 473        configurable: true
 474    }
 475});
 476Object.setPrototypeOf(NetworkError, Error);
 477&lt;/code&gt;&lt;/pre&gt;
 478&lt;p&gt;But hey, now we have a proper and reusable &lt;code&gt;NetworkError&lt;/code&gt;! Next, let’s make a proper and reusabe &lt;code&gt;query&lt;/code&gt; function that deals with &lt;code&gt;fetch&lt;/code&gt; for us:&lt;/p&gt;
 479&lt;pre&gt;&lt;code&gt;async function query(endpoint) {
 480    const res = await fetch(endpoint, {
 481        // if we ever use cookies, this is important
 482        credentials: 'include'
 483    });
 484    if (res.ok) {
 485        return await res.json();
 486    } else {
 487        throw new NetworkError(await res.text(), res.status);
 488    }
 489}
 490&lt;/code&gt;&lt;/pre&gt;
 491&lt;p&gt;At last, we can query our web server. The export here tells Svelte that this function should be visible to outer modules (public) as opposed to being private:&lt;/p&gt;
 492&lt;pre&gt;&lt;code&gt;export function get_area_info(long, lat, distance) {
 493    return query(`/rest/get-area-info?long=${long}&amp;amp;lat=${lat}&amp;amp;distance=${distance}`);
 494}
 495&lt;/code&gt;&lt;/pre&gt;
 496&lt;p&gt;The attentive reader will have noticed that &lt;code&gt;query&lt;/code&gt; is &lt;code&gt;async&lt;/code&gt;, but &lt;code&gt;get_area_info&lt;/code&gt; is not. This is intentional, because we don’t need to &lt;code&gt;await&lt;/code&gt; for anything inside of it. We can just return the &lt;code&gt;[Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)&lt;/code&gt; that &lt;code&gt;query&lt;/code&gt; created and let the caller &lt;code&gt;await&lt;/code&gt; it as they see fit. The &lt;code&gt;await&lt;/code&gt; here would have been redundant.&lt;/p&gt;
 497&lt;p&gt;For those of you who don’t know what a JavaScript promise is, think of it as an object that represents «an eventual result». The result may not be there yet, but we promised it will be present in the future, and we can &lt;code&gt;await&lt;/code&gt; for it. You can also find the same concept in other languages like Python under a different name, such as &lt;a href=&quot;https://docs.python.org/3/library/asyncio-future.html#asyncio.Future&quot;&gt;&lt;code&gt;Future&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;
 498&lt;h3 id=&quot;map_component&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#map_component&quot;&gt;¶&lt;/a&gt;Map component&lt;/h3&gt;
 499&lt;p&gt;In Svelte, we can define self-contained components that are issolated from the rest. This makes it really easy to create a modular application. Think of a Svelte component as your own HTML tag, which you can customize however you want, building upon the already-existing components HTML has to offer.&lt;/p&gt;
 500&lt;p&gt;The main thing that our map needs to do is render the map as an image and overlay the selection area as the user hovers the map with their mouse. We could render the image in the canvas itself, but instead we’ll use the HTML &lt;code&gt;&amp;lt;img&amp;gt;&lt;/code&gt; tag for that and put a transparent &lt;code&gt;&amp;lt;canvas&amp;gt;&lt;/code&gt; on top with some CSS. This should make it cheaper and easier to render things on the canvas.&lt;/p&gt;
 501&lt;p&gt;The &lt;code&gt;Map&lt;/code&gt; component will thus render as the user moves the mouse over it, and produce an event when they click so that whatever component is using a &lt;code&gt;Map&lt;/code&gt; knows that it was clicked. Here’s the final CSS and HTML:&lt;/p&gt;
 502&lt;pre&gt;&lt;code&gt;&amp;lt;style&amp;gt;
 503div {
 504    position: relative;
 505}
 506canvas {
 507    position: absolute;
 508    left: 0;
 509    top: 0;
 510    cursor: crosshair;
 511}
 512&amp;lt;/style&amp;gt;
 513
 514&amp;lt;div&amp;gt;
 515    &amp;lt;img bind:this={img} on:load={handleLoad} {height} src=&amp;quot;caceres-municipality.svg&amp;quot; alt=&amp;quot;Cáceres (municipality)&amp;quot;/&amp;gt;
 516    &amp;lt;canvas
 517        bind:this={canvas}
 518        on:mousemove={handleMove}
 519        on:wheel={handleWheel}
 520        on:mouseup={handleClick}/&amp;gt;
 521&amp;lt;/div&amp;gt;
 522&lt;/code&gt;&lt;/pre&gt;
 523&lt;p&gt;We hardcode a map source here, but ideally this would be provided by the server. The project is already complex enough, so we tried to avoid more complexity than necessary.&lt;/p&gt;
 524&lt;p&gt;We bind the tags to some variables declared in the JavaScript code of the component, along with some functions and parameters to let the users of &lt;code&gt;Map&lt;/code&gt; customize it just a little.&lt;/p&gt;
 525&lt;p&gt;Here’s the gist of the JavaScript code:&lt;/p&gt;
 526&lt;pre&gt;&lt;code&gt;&amp;lt;script&amp;gt;
 527    import { createEventDispatcher, onMount } from 'svelte';
 528
 529    export let height = 200;
 530
 531    const dispatch = createEventDispatcher();
 532
 533    let img;
 534    let canvas;
 535
 536    const LONG_WEST = -6.426881;
 537    const LONG_EAST = -6.354143;
 538    const LAT_NORTH = 39.500064;
 539    const LAT_SOUTH = 39.443201;
 540
 541    let x = 0;
 542    let y = 0;
 543    let clickInfo = null; // [x, y, radius]
 544    let radiusDelta = 0.005 * height;
 545    let maxRadius = 0.2 * height;
 546    let minRadius = 0.01 * height;
 547    let radius = 0.05 * height;
 548
 549    function handleLoad() {
 550        canvas.width = img.width;
 551        canvas.height = img.height;
 552    }
 553
 554    function handleMove(event) {
 555        const { left, top } = this.getBoundingClientRect();
 556        x = Math.round(event.clientX - left);
 557        y = Math.round(event.clientY - top);
 558    }
 559
 560    function handleWheel(event) {
 561        if (event.deltaY &amp;lt; 0) {
 562            if (radius &amp;lt; maxRadius) {
 563                radius += radiusDelta;
 564            }
 565        } else {
 566            if (radius &amp;gt; minRadius) {
 567                radius -= radiusDelta;
 568            }
 569        }
 570        event.preventDefault();
 571    }
 572
 573    function handleClick(event) {
 574        dispatch('click', {
 575            // the real code here maps the x/y/radius values to the right range, here omitted
 576            x: ...,
 577            y: ...,
 578            radius: ...,
 579        });
 580    }
 581
 582    onMount(() =&amp;gt; {
 583        const ctx = canvas.getContext('2d');
 584        let frame;
 585
 586        (function loop() {
 587            frame = requestAnimationFrame(loop);
 588
 589            // the real code renders mouse area/selection, here omitted for brevity
 590            ...
 591        }());
 592
 593        return () =&amp;gt; {
 594            cancelAnimationFrame(frame);
 595        };
 596    });
 597&amp;lt;/script&amp;gt;
 598&lt;/code&gt;&lt;/pre&gt;
 599&lt;p&gt;Let’s go through bit-by-bit:&lt;/p&gt;
 600&lt;ul&gt;
 601&lt;li&gt;We define a few variables and constants for later use in the final code.&lt;/li&gt;
 602&lt;li&gt;We define the handlers to react to mouse movement and clicks. On click, we dispatch an event to outer components.&lt;/li&gt;
 603&lt;li&gt;We setup the render loop with animation frames, and cancel the current frame appropriatedly if the component disappears.&lt;/li&gt;
 604&lt;/ul&gt;
 605&lt;h3 id=&quot;app_component&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#app_component&quot;&gt;¶&lt;/a&gt;App component&lt;/h3&gt;
 606&lt;p&gt;Time to put everything together! We wil include our function to make REST queries along with our &lt;code&gt;Map&lt;/code&gt; component to render things on screen.&lt;/p&gt;
 607&lt;pre&gt;&lt;code&gt;&amp;lt;script&amp;gt;
 608    import Map from './Map.svelte';
 609    import { get_area_info } from './rest.js'
 610    let selection = null;
 611    let area_info_promise = null;
 612    function handleMapSelection(event) {
 613        selection = event.detail;
 614        area_info_promise = get_area_info(selection.x, selection.y, selection.radius);
 615    }
 616    function format_avg_noise(avg_noise) {
 617        if (avg_noise === null) {
 618            return '(no data)';
 619        } else {
 620            return `${avg_noise.toFixed(2)} dB`;
 621        }
 622    }
 623&amp;lt;/script&amp;gt;
 624
 625&amp;lt;div class=&amp;quot;container-fluid&amp;quot;&amp;gt;
 626    &amp;lt;div class=&amp;quot;row&amp;quot;&amp;gt;
 627        &amp;lt;div class=&amp;quot;col-3&amp;quot; style=&amp;quot;max-width: 300em;&amp;quot;&amp;gt;
 628            &amp;lt;div class=&amp;quot;text-center&amp;quot;&amp;gt;
 629                &amp;lt;h1&amp;gt;Caceres Data Consultory&amp;lt;/h1&amp;gt;
 630            &amp;lt;/div&amp;gt;
 631            &amp;lt;Map height={400} on:click={handleMapSelection}/&amp;gt;
 632            &amp;lt;div class=&amp;quot;text-center mt-4&amp;quot;&amp;gt;
 633                {#if selection === null}
 634                        &amp;lt;p class=&amp;quot;m-1 p-3 border border-bottom-0 bg-info text-white&amp;quot;&amp;gt;Click on the map to select the area you wish to see details for.&amp;lt;/p&amp;gt;
 635                {:else}
 636                        &amp;lt;h2 class=&amp;quot;bg-dark text-white&amp;quot;&amp;gt;Selected area&amp;lt;/h2&amp;gt;
 637                        &amp;lt;p&amp;gt;&amp;lt;b&amp;gt;Coordinates:&amp;lt;/b&amp;gt; ({selection.x}, {selection.y})&amp;lt;/p&amp;gt;
 638                        &amp;lt;p&amp;gt;&amp;lt;b&amp;gt;Radius:&amp;lt;/b&amp;gt; {selection.radius} meters&amp;lt;/p&amp;gt;
 639                {/if}
 640            &amp;lt;/div&amp;gt;
 641        &amp;lt;/div&amp;gt;
 642        &amp;lt;div class=&amp;quot;col-sm-4&amp;quot;&amp;gt;
 643            &amp;lt;div class=&amp;quot;row&amp;quot;&amp;gt;
 644            {#if area_info_promise !== null}
 645                {#await area_info_promise}
 646                    &amp;lt;p&amp;gt;Fetching area information…&amp;lt;/p&amp;gt;
 647                {:then area_info}
 648                    &amp;lt;div class=&amp;quot;col&amp;quot;&amp;gt;
 649                        &amp;lt;div class=&amp;quot;text-center&amp;quot;&amp;gt;
 650                            &amp;lt;h2 class=&amp;quot;m-1 bg-dark text-white&amp;quot;&amp;gt;Area information&amp;lt;/h2&amp;gt;
 651                            &amp;lt;ul class=&amp;quot;list-unstyled&amp;quot;&amp;gt;
 652                                &amp;lt;li&amp;gt;There are &amp;lt;b&amp;gt;{area_info.tree_count} trees &amp;lt;/b&amp;gt; within the area&amp;lt;/li&amp;gt;
 653                                &amp;lt;li&amp;gt;The &amp;lt;b&amp;gt;average noise&amp;lt;/b&amp;gt; is &amp;lt;b&amp;gt;{format_avg_noise(area_info.avg_noise)}&amp;lt;/b&amp;gt;&amp;lt;/li&amp;gt;
 654                                &amp;lt;li&amp;gt;There are &amp;lt;b&amp;gt;{area_info.census_count} persons &amp;lt;/b&amp;gt; within the area&amp;lt;/li&amp;gt;
 655                            &amp;lt;/ul&amp;gt;
 656                        &amp;lt;/div&amp;gt;
 657                        {#if area_info.trees_per_type.length &amp;gt; 0} 
 658                            &amp;lt;div class=&amp;quot;text-center&amp;quot;&amp;gt;
 659                                &amp;lt;h2 class=&amp;quot;m-1 bg-dark text-white&amp;quot;&amp;gt;Tree count per type&amp;lt;/h2&amp;gt;
 660                            &amp;lt;/div&amp;gt;
 661                            &amp;lt;ul class=&amp;quot;list-group&amp;quot;&amp;gt;
 662                                {#each area_info.trees_per_type as [type, count]}
 663                                    &amp;lt;li class=&amp;quot;list-group-item&amp;quot;&amp;gt;{type} &amp;lt;span class=&amp;quot;badge badge-dark float-right&amp;quot;&amp;gt;{count}&amp;lt;/span&amp;gt;&amp;lt;/li&amp;gt;
 664                                {/each}
 665                            &amp;lt;/ul&amp;gt;
 666                        {/if}
 667                    &amp;lt;/div&amp;gt;
 668                {:catch error}
 669                    &amp;lt;p&amp;gt;Failed to fetch area information: {error.message}&amp;lt;/p&amp;gt;
 670                {/await}
 671            {/if}
 672            &amp;lt;/div&amp;gt;
 673        &amp;lt;/div&amp;gt;
 674    &amp;lt;/div&amp;gt;
 675&amp;lt;/div&amp;gt;
 676&lt;/code&gt;&lt;/pre&gt;
 677&lt;ul&gt;
 678&lt;li&gt;We import the &lt;code&gt;Map&lt;/code&gt; component and REST function so we can use them.&lt;/li&gt;
 679&lt;li&gt;We define a listener for the events that the &lt;code&gt;Map&lt;/code&gt; produces. Such event will trigger a REST call to the server and save the result in a promise used later.&lt;/li&gt;
 680&lt;li&gt;We’re using Bootstrap for the layout because it’s a lot easier. In the body we add our &lt;code&gt;Map&lt;/code&gt; and another column to show the selection information.&lt;/li&gt;
 681&lt;li&gt;We make use of Svelte’s &lt;code&gt;{#await}&lt;/code&gt; to nicely notify the user when the call is being made, when it was successful, and when it failed. If it’s successful, we display the info.&lt;/li&gt;
 682&lt;/ul&gt;
 683&lt;h2 id=&quot;results&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#results&quot;&gt;¶&lt;/a&gt;Results&lt;/h2&gt;
 684&lt;p&gt;Lo and behold, watch our application run!&lt;/p&gt;
 685&lt;p&gt;&lt;video controls=&quot;controls&quot; src=&quot;sr-2020-04-14_09-28-25.mp4&quot;&gt;&lt;/video&gt;&lt;/p&gt;
 686&lt;p&gt;In this video you can see our application running, but let’s describe what is happening in more detail.&lt;/p&gt;
 687&lt;p&gt;When the application starts running (by opening it in your web browser of choice), you can see a map with the town of Cáceres. Then you, the user, can click to retrieve the information within the selected area.&lt;/p&gt;
 688&lt;p&gt;It is important to note that one can make the selection area larger or smaller by trying to scroll up or down, respectively.&lt;/p&gt;
 689&lt;p&gt;Once an area is selected, it is colored green in order to let the user know which area they have selected. Under the map, the selected coordinates and the radius (in meters) is also shown for the curious. At the right side the information concerning the selected area is shown, such as the number of trees, the average noise and the number of persons. If there are trees in the area, the application also displays the trees per type, sorted by the number of trees.&lt;/p&gt;
 690&lt;h2 id=&quot;download&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#download&quot;&gt;¶&lt;/a&gt;Download&lt;/h2&gt;
 691&lt;p&gt;We hope you enjoyed reading this post as much as we enjoyed writing it! Feel free to download the final project and play around with it. Maybe you can adapt it for even more interesting purposes!&lt;/p&gt;
 692&lt;p&gt;&lt;em&gt;download removed&lt;/em&gt;&lt;/p&gt;
 693&lt;p&gt;To run the above code:&lt;/p&gt;
 694&lt;ol&gt;
 695&lt;li&gt;Unzip the downloaded file.&lt;/li&gt;
 696&lt;li&gt;Make a copy of &lt;code&gt;example-server-config.ini&lt;/code&gt; and rename it to &lt;code&gt;server-config.ini&lt;/code&gt;, then edit the file to suit your needs.&lt;/li&gt;
 697&lt;li&gt;Run the server with &lt;code&gt;python -m server&lt;/code&gt;.&lt;/li&gt;
 698&lt;li&gt;Open &lt;a href=&quot;http://localhost:9000&quot;&gt;localhost:9000&lt;/a&gt; in your web browser (or whatever port you chose) and enjoy!&lt;/li&gt;
 699&lt;/ol&gt;
 700&lt;/main&gt;
 701&lt;/body&gt;
 702&lt;/html&gt;
 703 </content></entry><entry><title>MongoDB: an Introduction</title><id>dist/mongodb-an-introduction/index.html</id><updated>2020-04-07T22:00:00+00:00</updated><published>2020-03-04T23:00:00+00:00</published><summary>This is the first post in the MongoDB series, where we will introduce the MongoDB database system and take a look at its features and installation methods.</summary><content type="html" src="dist/mongodb-an-introduction/index.html">&lt;!DOCTYPE html&gt;
 704&lt;html&gt;
 705&lt;head&gt;
 706&lt;meta charset=&quot;utf-8&quot; /&gt;
 707&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
 708&lt;title&gt;MongoDB: an Introduction&lt;/title&gt;
 709&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
 710&lt;/head&gt;
 711&lt;body&gt;
 712&lt;main&gt;
 713&lt;p&gt;This is the first post in the MongoDB series, where we will introduce the MongoDB database system and take a look at its features and installation methods.&lt;/p&gt;
 714&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-05&lt;br&gt;
 715Modified 2020-04-08&lt;/div&gt;
 716&lt;p&gt;Other posts in this series:&lt;/p&gt;
 717&lt;ul&gt;
 718&lt;li&gt;&lt;a href=&quot;/blog/ribw/mongodb-an-introduction/&quot;&gt;MongoDB: an Introduction&lt;/a&gt; (this post)&lt;/li&gt;
 719&lt;li&gt;&lt;a href=&quot;/blog/ribw/mongodb-basic-operations-and-architecture/&quot;&gt;MongoDB: Basic Operations and Architecture&lt;/a&gt;&lt;/li&gt;
 720&lt;li&gt;&lt;a href=&quot;/blog/ribw/developing-a-python-application-for-mongodb/&quot;&gt;Developing a Python application for MongoDB&lt;/a&gt;&lt;/li&gt;
 721&lt;/ul&gt;
 722&lt;p&gt;This post is co-authored wih Classmate.&lt;/p&gt;
 723&lt;hr /&gt;
 724&lt;div class=&quot;image-container&quot;&gt;
 725&lt;img src=&quot;mongodb.png&quot; alt=&quot;NoSQL database – MongoDB – First delivery&quot; /&gt;
 726&lt;div class=&quot;image-caption&quot;&gt;&lt;/div&gt;
 727&lt;/div&gt;
 728&lt;p&gt;
 729&lt;h2 class=&quot;title&quot; id=&quot;purpose_of_technology&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#purpose_of_technology&quot;&gt;¶&lt;/a&gt;Purpose of technology&lt;/h2&gt;
 730&lt;p&gt;MongoDB is a &lt;strong&gt;general purpose, document-based, distributed database&lt;/strong&gt; built for modern application developers and for the cloud era, with the scalability and flexibility that you want with the querying and indexing that you need. It being a document database means it stores data in JSON-like documents.&lt;/p&gt;
 731&lt;p&gt;The Mongo team believes this is the most natural way to think about data, which is (they claim) much more expressive and powerful than the traditional row/column model, since programmers think in objects.&lt;/p&gt;
 732&lt;h2 id=&quot;how_it_works&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#how_it_works&quot;&gt;¶&lt;/a&gt;How it works&lt;/h2&gt;
 733&lt;p&gt;MongoDB’s architecture can be summarized as follows:&lt;/p&gt;
 734&lt;ul&gt;
 735&lt;li&gt;Document data model.&lt;/li&gt;
 736&lt;li&gt;Distributed systems design.&lt;/li&gt;
 737&lt;li&gt;Unified experience with freedom to run it anywhere.&lt;/li&gt;
 738&lt;/ul&gt;
 739&lt;p&gt;For a more in-depth explanation, MongoDB offers a &lt;a href=&quot;https://www.mongodb.com/collateral/mongodb-architecture-guide&quot;&gt;download to the MongoDB Architecture Guide&lt;/a&gt; with roughly ten pages worth of text.&lt;/p&gt;
 740&lt;p&gt;&lt;img src=&quot;knGHenfTGA4kzJb1PHmS9EQvtZl2QlhbIPN15M38m8fZfZf7ODwYfhf0Tltr.png&quot; alt=&quot;&quot; /&gt;
 741_ Overview of MongoDB’s architecture_&lt;/p&gt;
 742&lt;p&gt;Regarding usage, MongoDB comes with a really nice introduction along with JavaScript, Python, Java, C++ or C# code at our choice, which describes the steps necessary to make it work. Below we will describe a common workflow.&lt;/p&gt;
 743&lt;p&gt;First, we must &lt;strong&gt;connect&lt;/strong&gt; to a running MongoDB instance. Once the connection succeeds, we can access individual «collections», which we can think of as &lt;em&gt;tables&lt;/em&gt; where collections of data is stored.&lt;/p&gt;
 744&lt;p&gt;For instance, we could &lt;strong&gt;insert&lt;/strong&gt; an arbitrary JSON document into the &lt;code&gt;restaurants&lt;/code&gt; collection to store information about a restaurant.&lt;/p&gt;
 745&lt;p&gt;At any other point in time, we can &lt;strong&gt;query&lt;/strong&gt; these collections. The queries range from trivial, empty ones (which would retrieve all the documents and fields) to more rich and complex queries (for instance, using AND and OR operators, checking if data exists, and then looking for a value in a list).&lt;/p&gt;
 746&lt;p&gt;MongoDB also supports the creation of &lt;strong&gt;indices&lt;/strong&gt;, similar to those in other database systems. It allows for the creation of indices on any field or subfields.&lt;/p&gt;
 747&lt;p&gt;In Mongo, the &lt;strong&gt;aggregation pipeline&lt;/strong&gt; allows us to filter and analyze data based on a given set of criteria. For example, we could pull all the documents in the &lt;code&gt;restaurants&lt;/code&gt; collection that have a &lt;code&gt;category&lt;/code&gt; of &lt;code&gt;Bakery&lt;/code&gt; using the &lt;code&gt;$match&lt;/code&gt; operator. Then, we can group them by their star rating using the &lt;code&gt;$group&lt;/code&gt; operator. Using the accumulator operator, &lt;code&gt;$sum&lt;/code&gt;, we can see how many bakeries in our collection have each star rating.&lt;/p&gt;
 748&lt;h2 id=&quot;features&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#features&quot;&gt;¶&lt;/a&gt;Features&lt;/h2&gt;
 749&lt;p&gt;The features can be seen all over the place in their site, because it’s something they make a lot of emphasis on:&lt;/p&gt;
 750&lt;ul&gt;
 751&lt;li&gt;
 752&lt;p&gt;&lt;strong&gt;Easy development&lt;/strong&gt;, thanks to the document data model, something they claim to be «the best way to work with data».&lt;/p&gt;
 753&lt;/li&gt;
 754&lt;li&gt;
 755&lt;p&gt;Data is stored in flexible JSON-like documents.&lt;/p&gt;
 756&lt;/li&gt;
 757&lt;li&gt;
 758&lt;p&gt;This model directly maps to the objects in the application’s code.&lt;/p&gt;
 759&lt;/li&gt;
 760&lt;li&gt;
 761&lt;p&gt;Ad hoc queries, indexing, and real time aggregation provide powerful ways to access and analyze the data.&lt;/p&gt;
 762&lt;/li&gt;
 763&lt;li&gt;
 764&lt;p&gt;&lt;strong&gt;Powerful query language&lt;/strong&gt;, with a rich and expressive query language that allows filtering and sorting by any field, no matter how nested it may be within a document. The queries are themselves JSON, and thus easily composable.&lt;/p&gt;
 765&lt;/li&gt;
 766&lt;li&gt;
 767&lt;p&gt;&lt;strong&gt;Support for aggregations&lt;/strong&gt; and other modern use-cases such as geo-based search, graph search, and text search.&lt;/p&gt;
 768&lt;/li&gt;
 769&lt;li&gt;
 770&lt;p&gt;&lt;strong&gt;A distributed systems design&lt;/strong&gt;, which allows developers to intelligently put data where they want it. High availability, horizontal scaling, and geographic distribution are built in and easy to use.&lt;/p&gt;
 771&lt;/li&gt;
 772&lt;li&gt;
 773&lt;p&gt;&lt;strong&gt;A unified experience&lt;/strong&gt; with the freedom to run anywhere, which allows developers to future-proof their work and eliminate vendor lock-in.&lt;/p&gt;
 774&lt;/li&gt;
 775&lt;/ul&gt;
 776&lt;h2 id=&quot;corner_in_cap_theorem&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#corner_in_cap_theorem&quot;&gt;¶&lt;/a&gt;Corner in CAP theorem&lt;/h2&gt;
 777&lt;p&gt;MongoDB’s position in the CAP theorem (Consistency, Availability, Partition Tolerance) depends on the database and driver configurations, and the type of disaster.&lt;/p&gt;
 778&lt;ul&gt;
 779&lt;li&gt;With &lt;strong&gt;no partitions&lt;/strong&gt;, the main focus is &lt;strong&gt;CA&lt;/strong&gt;.&lt;/li&gt;
 780&lt;li&gt;If there are **partitions **but the system is &lt;strong&gt;strongly connected&lt;/strong&gt;, the main focus is &lt;strong&gt;AP&lt;/strong&gt;: non-synchronized writes from the old primary are ignored.&lt;/li&gt;
 781&lt;li&gt;If there are &lt;strong&gt;partitions&lt;/strong&gt; but the system is &lt;strong&gt;not strongly connected&lt;/strong&gt;, the main focus is &lt;strong&gt;CP&lt;/strong&gt;: only read access is provided to avoid inconsistencies.
 782The general consensus seems to be that Mongo is &lt;strong&gt;CP&lt;/strong&gt;.&lt;/li&gt;
 783&lt;/ul&gt;
 784&lt;h2 id=&quot;download&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#download&quot;&gt;¶&lt;/a&gt;Download&lt;/h2&gt;
 785&lt;p&gt;We will be using the apt-based installation.&lt;/p&gt;
 786&lt;p&gt;The Community version can be downloaded by anyone through &lt;a href=&quot;https://www.mongodb.com/download-center/community&quot;&gt;MongoDB Download Center&lt;/a&gt;, where one can choose the version, Operating System and Package.MongoDB also seems to be &lt;a href=&quot;https://packages.ubuntu.com/eoan/mongodb&quot;&gt;available in Ubuntu’s PPAs&lt;/a&gt;. &lt;/p&gt;
 787&lt;h2 id=&quot;installation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#installation&quot;&gt;¶&lt;/a&gt;Installation&lt;/h2&gt;
 788&lt;p&gt;We will be using an Ubuntu-based system, with apt available. To install MongoDB, we open a terminal and run the following command:&lt;/p&gt;
 789&lt;pre&gt;&lt;code&gt;apt install mongodb
 790&lt;/code&gt;&lt;/pre&gt;
 791&lt;p&gt;After confirming that we do indeed want to install the package, we should be able to run the following command to verify that the installation was successful:&lt;/p&gt;
 792&lt;pre&gt;&lt;code&gt;mongod --version
 793&lt;/code&gt;&lt;/pre&gt;
 794&lt;p&gt;The output should be similar to the following: &lt;/p&gt;
 795&lt;pre&gt;&lt;code&gt;db version v4.0.16
 796git version: 2a5433168a53044cb6b4fa8083e4cfd7ba142221
 797OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
 798allocator: tcmalloc
 799modules: none
 800build environment:
 801	distmod: ubuntu1804
 802	distarch: x86_64
 803	target_arch: x86_64
 804&lt;/code&gt;&lt;/pre&gt;
 805&lt;h2 id=&quot;references&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#references&quot;&gt;¶&lt;/a&gt;References&lt;/h2&gt;
 806&lt;ul&gt;
 807&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/&quot;&gt;MongoDB’s official site&lt;/a&gt;&lt;/li&gt;
 808&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/what-is-mongodb&quot;&gt;What is MongoDB?&lt;/a&gt;&lt;/li&gt;
 809&lt;li&gt;&lt;a href=&quot;https://www.mongodb.com/mongodb-architecture&quot;&gt;MongoDB Architecture&lt;/a&gt;&lt;/li&gt;
 810&lt;li&gt;&lt;a href=&quot;https://stackoverflow.com/q/11292215/4759433&quot;&gt;Where does mongodb stand in the CAP theorem?&lt;/a&gt;&lt;/li&gt;
 811&lt;li&gt;&lt;a href=&quot;https://medium.com/@bikas.katwal10/mongodb-vs-cassandra-vs-rdbms-where-do-they-stand-in-the-cap-theorem-1bae779a7a15&quot;&gt;What is the CAP Theorem? MongoDB vs Cassandra vs RDBMS, where do they stand in the CAP theorem?&lt;/a&gt;&lt;/li&gt;
 812&lt;li&gt;&lt;a href=&quot;https://www.quora.com/Why-doesnt-MongoDB-have-availability-in-the-CAP-theorem&quot;&gt;Why doesn’t MongoDB have availability in the CAP theorem?&lt;/a&gt;&lt;/li&gt;
 813&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/installation/&quot;&gt;Install MongoDB&lt;/a&gt;&lt;/li&gt;
 814&lt;/ul&gt;
 815&lt;/main&gt;
 816&lt;/body&gt;
 817&lt;/html&gt;
 818 </content></entry><entry><title>MongoDB: Basic Operations and Architecture</title><id>dist/mongodb-basic-operations-and-architecture/index.html</id><updated>2020-04-07T22:00:00+00:00</updated><published>2020-03-04T23:00:00+00:00</published><summary>This is the second post in the MongoDB series, where we will take a look at the </summary><content type="html" src="dist/mongodb-basic-operations-and-architecture/index.html">&lt;!DOCTYPE html&gt;
 819&lt;html&gt;
 820&lt;head&gt;
 821&lt;meta charset=&quot;utf-8&quot; /&gt;
 822&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
 823&lt;title&gt;MongoDB: Basic Operations and Architecture&lt;/title&gt;
 824&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
 825&lt;/head&gt;
 826&lt;body&gt;
 827&lt;main&gt;
 828&lt;p&gt;This is the second post in the MongoDB series, where we will take a look at the &lt;a href=&quot;https://stackify.com/what-are-crud-operations/&quot;&gt;CRUD operations&lt;/a&gt; they support, the data model and architecture used.&lt;/p&gt;
 829&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-05&lt;br&gt;
 830Modified 2020-04-08&lt;/div&gt;
 831&lt;p&gt;Other posts in this series:&lt;/p&gt;
 832&lt;ul&gt;
 833&lt;li&gt;&lt;a href=&quot;/blog/ribw/mongodb-an-introduction/&quot;&gt;MongoDB: an Introduction&lt;/a&gt;&lt;/li&gt;
 834&lt;li&gt;&lt;a href=&quot;/blog/ribw/mongodb-basic-operations-and-architecture/&quot;&gt;MongoDB: Basic Operations and Architecture&lt;/a&gt; (this post)&lt;/li&gt;
 835&lt;li&gt;&lt;a href=&quot;/blog/ribw/developing-a-python-application-for-mongodb/&quot;&gt;Developing a Python application for MongoDB&lt;/a&gt;&lt;/li&gt;
 836&lt;/ul&gt;
 837&lt;p&gt;This post is co-authored wih Classmate, and in it we will take an explorative approach using the &lt;code&gt;mongo&lt;/code&gt; command line shell to execute commands against the database. It even has TAB auto-completion, which is awesome!&lt;/p&gt;
 838&lt;hr /&gt;
 839&lt;p&gt;Before creating any documents, we first need to create somewhere for the documents to be in. And before we create anything, the database has to be running, so let’s do that first. If we don’t have a service installed, we can run the &lt;code&gt;mongod&lt;/code&gt; command ourselves in some local folder to make things easier:&lt;/p&gt;
 840&lt;pre&gt;&lt;code&gt;$ mkdir -p mongo-database
 841$ mongod --dbpath mongo-database
 842&lt;/code&gt;&lt;/pre&gt;
 843&lt;p&gt;Just like that, we will have Mongo running. Now, let’s connect to it using the &lt;code&gt;mongo&lt;/code&gt; command in another terminal (don’t close the terminal where the server is running, we need it!). By default, it connects to localhost, which is just what we need.&lt;/p&gt;
 844&lt;pre&gt;&lt;code&gt;$ mongo
 845&lt;/code&gt;&lt;/pre&gt;
 846&lt;h2 class=&quot;title&quot; id=&quot;create&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#create&quot;&gt;¶&lt;/a&gt;Create&lt;/h2&gt;
 847&lt;h3 id=&quot;create_a_database&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#create_a_database&quot;&gt;¶&lt;/a&gt;Create a database&lt;/h3&gt;
 848&lt;p&gt;Let’s list the databases:&lt;/p&gt;
 849&lt;pre&gt;&lt;code&gt;&amp;gt; show databases
 850admin   0.000GB
 851config  0.000GB
 852local   0.000GB
 853&lt;/code&gt;&lt;/pre&gt;
 854&lt;p&gt;Oh, how interesting! There’s already some databases, even though we just created the directory where Mongo will store everything. However, they seem empty, which make sense.&lt;/p&gt;
 855&lt;p&gt;Creating a new database is done by &lt;code&gt;use&lt;/code&gt;-ing a name that doesn’t exist. Let’s call our new database «helloworld».&lt;/p&gt;
 856&lt;pre&gt;&lt;code&gt;&amp;gt; use helloworld
 857switched to db helloworld
 858&lt;/code&gt;&lt;/pre&gt;
 859&lt;p&gt;Good! Now the «local variable» called &lt;code&gt;db&lt;/code&gt; points to our &lt;code&gt;helloworld&lt;/code&gt; database.&lt;/p&gt;
 860&lt;pre&gt;&lt;code&gt;&amp;gt; db
 861helloworld
 862&lt;/code&gt;&lt;/pre&gt;
 863&lt;p&gt;What happens if we print the databases again? Surely our new database will show up now…&lt;/p&gt;
 864&lt;pre&gt;&lt;code&gt;&amp;gt; show databases
 865admin   0.000GB
 866config  0.000GB
 867local   0.000GB
 868&lt;/code&gt;&lt;/pre&gt;
 869&lt;p&gt;…maybe not! It seems Mongo won’t create the database until we create some collections and documents in it. Databases contain collections, and inside collections (which you can think of as tables) we can insert new documents (which you can think of as rows). Like in many programming languages, the dot operator is used to access these «members».&lt;/p&gt;
 870&lt;h3 id=&quot;create_a_document&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#create_a_document&quot;&gt;¶&lt;/a&gt;Create a document&lt;/h3&gt;
 871&lt;p&gt;Let’s add a new greeting into the &lt;code&gt;greetings&lt;/code&gt; collection:&lt;/p&gt;
 872&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.insert({message: &amp;quot;¡Bienvenido!&amp;quot;, lang: &amp;quot;es&amp;quot;})
 873WriteResult({ &amp;quot;nInserted&amp;quot; : 1 })
 874
 875&amp;gt; show collections
 876greetings
 877
 878&amp;gt; show databases
 879admin       0.000GB
 880config      0.000GB
 881helloworld  0.000GB
 882local       0.000GB
 883&lt;/code&gt;&lt;/pre&gt;
 884&lt;p&gt;That looks promising! We can also see our new &lt;code&gt;helloworld&lt;/code&gt; database also shows up. The Mongo shell actually works on JavaScript-like code, which is why we can use a variant of JSON (BSON) to insert documents (note the lack of quotes around the keys, convenient!).&lt;/p&gt;
 885&lt;p&gt;The &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.collection.insert/index.html&quot;&gt;&lt;code&gt;insert&lt;/code&gt;&lt;/a&gt; method actually supports a list of documents, and by default Mongo will assign a unique identifier to each. If we don’t want that though, all we have to do is add the &lt;code&gt;_id&lt;/code&gt; key to our documents.&lt;/p&gt;
 886&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.insert([
 887... {message: &amp;quot;Welcome!&amp;quot;, lang: &amp;quot;en&amp;quot;},
 888... {message: &amp;quot;Bonjour!&amp;quot;, lang: &amp;quot;fr&amp;quot;},
 889... ])
 890BulkWriteResult({
 891    &amp;quot;writeErrors&amp;quot; : [ ],
 892    &amp;quot;writeConcernErrors&amp;quot; : [ ],
 893    &amp;quot;nInserted&amp;quot; : 2,
 894    &amp;quot;nUpserted&amp;quot; : 0,
 895    &amp;quot;nMatched&amp;quot; : 0,
 896    &amp;quot;nModified&amp;quot; : 0,
 897    &amp;quot;nRemoved&amp;quot; : 0,
 898    &amp;quot;upserted&amp;quot; : [ ]
 899})
 900&lt;/code&gt;&lt;/pre&gt;
 901&lt;h3 id=&quot;create_a_collection&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#create_a_collection&quot;&gt;¶&lt;/a&gt;Create a collection&lt;/h3&gt;
 902&lt;p&gt;In this example, we created the collection &lt;code&gt;greetings&lt;/code&gt; implicitly, but behind the scenes Mongo made a call to &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.createCollection/&quot;&gt;&lt;code&gt;createCollection&lt;/code&gt;&lt;/a&gt;. Let’s do just that:&lt;/p&gt;
 903&lt;pre&gt;&lt;code&gt;&amp;gt; db.createCollection(&amp;quot;goodbyes&amp;quot;)
 904{ &amp;quot;ok&amp;quot; : 1 }
 905
 906&amp;gt; show collections
 907goodbyes
 908greetings
 909&lt;/code&gt;&lt;/pre&gt;
 910&lt;p&gt;The method actually has a default parameter to configure other options, like the maximum size of the collection or maximum amount of documents in it, validation-related options, and so on. These are all described in more details in the documentation.&lt;/p&gt;
 911&lt;h2 id=&quot;read&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#read&quot;&gt;¶&lt;/a&gt;Read&lt;/h2&gt;
 912&lt;p&gt;To read the contents of a document, we have to &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.collection.find/index.html&quot;&gt;&lt;code&gt;find&lt;/code&gt;&lt;/a&gt; it.&lt;/p&gt;
 913&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.find()
 914{ &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e74829a0659f802b15f18dd&amp;quot;), &amp;quot;message&amp;quot; : &amp;quot;¡Bienvenido!&amp;quot;, &amp;quot;lang&amp;quot; : &amp;quot;es&amp;quot; }
 915{ &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e7487b90659f802b15f18de&amp;quot;), &amp;quot;message&amp;quot; : &amp;quot;Welcome!&amp;quot;, &amp;quot;lang&amp;quot; : &amp;quot;en&amp;quot; }
 916{ &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e7487b90659f802b15f18df&amp;quot;), &amp;quot;message&amp;quot; : &amp;quot;Bonjour!&amp;quot;, &amp;quot;lang&amp;quot; : &amp;quot;fr&amp;quot; }
 917&lt;/code&gt;&lt;/pre&gt;
 918&lt;p&gt;That’s a bit unreadable for my taste, can we make it more &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/cursor.pretty/index.html&quot;&gt;&lt;code&gt;pretty&lt;/code&gt;&lt;/a&gt;?&lt;/p&gt;
 919&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.find().pretty()
 920{
 921    &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e74829a0659f802b15f18dd&amp;quot;),
 922    &amp;quot;message&amp;quot; : &amp;quot;¡Bienvenido!&amp;quot;,
 923    &amp;quot;lang&amp;quot; : &amp;quot;es&amp;quot;
 924}
 925{
 926    &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e7487b90659f802b15f18de&amp;quot;),
 927    &amp;quot;message&amp;quot; : &amp;quot;Welcome!&amp;quot;,
 928    &amp;quot;lang&amp;quot; : &amp;quot;en&amp;quot;
 929}
 930{
 931    &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e7487b90659f802b15f18df&amp;quot;),
 932    &amp;quot;message&amp;quot; : &amp;quot;Bonjour!&amp;quot;,
 933    &amp;quot;lang&amp;quot; : &amp;quot;fr&amp;quot;
 934}
 935&lt;/code&gt;&lt;/pre&gt;
 936&lt;p&gt;Gorgeous! We can clearly see Mongo created an identifier for us automatically. The queries are also JSON, and support a bunch of operators (prefixed by &lt;code&gt;$&lt;/code&gt;), known as &lt;a href=&quot;https://docs.mongodb.com/manual/reference/operator/query/&quot;&gt;Query Selectors&lt;/a&gt;. Here’s a few:&lt;/p&gt;
 937&lt;table&gt;
 938 &lt;thead&gt;
 939  &lt;tr&gt;
 940   &lt;th&gt;
 941    Operation
 942   &lt;/th&gt;
 943   &lt;th&gt;
 944    Syntax
 945   &lt;/th&gt;
 946   &lt;th&gt;
 947    RDBMS equivalent
 948   &lt;/th&gt;
 949  &lt;/tr&gt;
 950 &lt;/thead&gt;
 951 &lt;tbody&gt;
 952  &lt;tr&gt;
 953   &lt;td&gt;
 954    Equals
 955   &lt;/td&gt;
 956   &lt;td&gt;
 957    &lt;code&gt;
 958     {key: {$eq: value}}
 959    &lt;/code&gt;
 960    &lt;br/&gt;
 961    Shorthand:
 962    &lt;code&gt;
 963     {key: value}
 964    &lt;/code&gt;
 965   &lt;/td&gt;
 966   &lt;td&gt;
 967    &lt;code&gt;
 968     where key = value
 969    &lt;/code&gt;
 970   &lt;/td&gt;
 971  &lt;/tr&gt;
 972  &lt;tr&gt;
 973   &lt;td&gt;
 974    Less Than
 975   &lt;/td&gt;
 976   &lt;td&gt;
 977    &lt;code&gt;
 978     {key: {$lte: value}}
 979    &lt;/code&gt;
 980   &lt;/td&gt;
 981   &lt;td&gt;
 982    &lt;code&gt;
 983     where key &amp;lt; value
 984    &lt;/code&gt;
 985   &lt;/td&gt;
 986  &lt;/tr&gt;
 987  &lt;tr&gt;
 988   &lt;td&gt;
 989    Less Than or Equal
 990   &lt;/td&gt;
 991   &lt;td&gt;
 992    &lt;code&gt;
 993     {key: {$lt: value}}
 994    &lt;/code&gt;
 995   &lt;/td&gt;
 996   &lt;td&gt;
 997    &lt;code&gt;
 998     where key &amp;lt;= value
 999    &lt;/code&gt;
1000   &lt;/td&gt;
1001  &lt;/tr&gt;
1002  &lt;tr&gt;
1003   &lt;td&gt;
1004    Greater Than
1005   &lt;/td&gt;
1006   &lt;td&gt;
1007    &lt;code&gt;
1008     {key: {$gt: value}}
1009    &lt;/code&gt;
1010   &lt;/td&gt;
1011   &lt;td&gt;
1012    &lt;code&gt;
1013     where key &amp;gt; value
1014    &lt;/code&gt;
1015   &lt;/td&gt;
1016  &lt;/tr&gt;
1017  &lt;tr&gt;
1018   &lt;td&gt;
1019    Greater Than or Equal
1020   &lt;/td&gt;
1021   &lt;td&gt;
1022    &lt;code&gt;
1023     {key: {$gte: value}}
1024    &lt;/code&gt;
1025   &lt;/td&gt;
1026   &lt;td&gt;
1027    &lt;code&gt;
1028     where key &amp;gt;= value
1029    &lt;/code&gt;
1030   &lt;/td&gt;
1031  &lt;/tr&gt;
1032  &lt;tr&gt;
1033   &lt;td&gt;
1034    Not Equal
1035   &lt;/td&gt;
1036   &lt;td&gt;
1037    &lt;code&gt;
1038     {key: {$ne: value}}
1039    &lt;/code&gt;
1040   &lt;/td&gt;
1041   &lt;td&gt;
1042    &lt;code&gt;
1043     where key != value
1044    &lt;/code&gt;
1045   &lt;/td&gt;
1046  &lt;/tr&gt;
1047  &lt;tr&gt;
1048   &lt;td&gt;
1049    And
1050   &lt;/td&gt;
1051   &lt;td&gt;
1052    &lt;code&gt;
1053     {$and: [{k1: v1}, {k2: v2}]}
1054    &lt;/code&gt;
1055   &lt;/td&gt;
1056   &lt;td&gt;
1057    &lt;code&gt;
1058     where k1 = v1 and k2 = v2
1059    &lt;/code&gt;
1060   &lt;/td&gt;
1061  &lt;/tr&gt;
1062  &lt;tr&gt;
1063   &lt;td&gt;
1064    Or
1065   &lt;/td&gt;
1066   &lt;td&gt;
1067    &lt;code&gt;
1068     {$or: [{k1: v1}, {k2: v2}]}
1069    &lt;/code&gt;
1070   &lt;/td&gt;
1071   &lt;td&gt;
1072    &lt;code&gt;
1073     where k1 = v1 or k2 = v2
1074    &lt;/code&gt;
1075   &lt;/td&gt;
1076  &lt;/tr&gt;
1077 &lt;/tbody&gt;
1078&lt;/table&gt;
1079&lt;p&gt;The operations all do what you would expect them to do, and their names are really intuitive. Aggregating operations with &lt;code&gt;$and&lt;/code&gt; or &lt;code&gt;$or&lt;/code&gt; can be done anywhere in the query, nested any level deep.&lt;/p&gt;
1080&lt;h2 id=&quot;update&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#update&quot;&gt;¶&lt;/a&gt;Update&lt;/h2&gt;
1081&lt;p&gt;Updating a document can be done by using &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.collection.save/index.html&quot;&gt;&lt;code&gt;save&lt;/code&gt;&lt;/a&gt; on an already-existing document (that is, the document we want to save has &lt;code&gt;_id&lt;/code&gt; and it’s in the collection already). If the document is not in the collection yet, this method will create it.&lt;/p&gt;
1082&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.save({_id: ObjectId(&amp;quot;5e74829a0659f802b15f18dd&amp;quot;), message: &amp;quot;¡Bienvenido, humano!&amp;quot;, &amp;quot;lang&amp;quot; : &amp;quot;es&amp;quot;})
1083WriteResult({ &amp;quot;nMatched&amp;quot; : 1, &amp;quot;nUpserted&amp;quot; : 0, &amp;quot;nModified&amp;quot; : 1 })
1084
1085&amp;gt; db.greetings.find({lang: &amp;quot;es&amp;quot;})
1086{ &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e74829a0659f802b15f18dd&amp;quot;), &amp;quot;message&amp;quot; : &amp;quot;¡Bienvenido, humano!&amp;quot;, &amp;quot;lang&amp;quot; : &amp;quot;es&amp;quot; }
1087&lt;/code&gt;&lt;/pre&gt;
1088&lt;p&gt;Alternatively, the &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.collection.update/index.html&quot;&gt;&lt;code&gt;update&lt;/code&gt;&lt;/a&gt; method takes a query and new value.&lt;/p&gt;
1089&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.update({lang: &amp;quot;en&amp;quot;}, {$set: {message: &amp;quot;Welcome, human!&amp;quot;}})
1090WriteResult({ &amp;quot;nMatched&amp;quot; : 1, &amp;quot;nUpserted&amp;quot; : 0, &amp;quot;nModified&amp;quot; : 1 })
1091
1092&amp;gt; db.greetings.find({lang: &amp;quot;en&amp;quot;})
1093{ &amp;quot;_id&amp;quot; : ObjectId(&amp;quot;5e7487b90659f802b15f18de&amp;quot;), &amp;quot;message&amp;quot; : &amp;quot;Welcome, human!&amp;quot;, &amp;quot;lang&amp;quot; : &amp;quot;en&amp;quot; }
1094&lt;/code&gt;&lt;/pre&gt;
1095&lt;h2 id=&quot;indexing&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#indexing&quot;&gt;¶&lt;/a&gt;Indexing&lt;/h2&gt;
1096&lt;p&gt;Creating an index is done with &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.collection.createIndex/index.html&quot;&gt;&lt;code&gt;createIndex&lt;/code&gt;&lt;/a&gt;:&lt;/p&gt;
1097&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.createIndex({lang: +1})
1098{
1099    &amp;quot;createdCollectionAutomatically&amp;quot; : false,
1100    &amp;quot;numIndexesBefore&amp;quot; : 1,
1101    &amp;quot;numIndexesAfter&amp;quot; : 2,
1102    &amp;quot;ok&amp;quot; : 1
1103}
1104&lt;/code&gt;&lt;/pre&gt;
1105&lt;p&gt;Here, we create an ascending index on the lang key. Descending order is done with &lt;code&gt;-1&lt;/code&gt;. Now a query for &lt;code&gt;lang&lt;/code&gt; in our three documents will be fast… well maybe iteration over three documents was faster than an index.&lt;/p&gt;
1106&lt;h2 id=&quot;delete&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#delete&quot;&gt;¶&lt;/a&gt;Delete&lt;/h2&gt;
1107&lt;h3 id=&quot;delete_a_document&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#delete_a_document&quot;&gt;¶&lt;/a&gt;Delete a document&lt;/h3&gt;
1108&lt;p&gt;I have to confess, I can’t talk French. I learnt it long ago and it’s long forgotten, so let’s remove the translation I copied online from our greetings with &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.collection.remove/index.html&quot;&gt;&lt;code&gt;remove&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;
1109&lt;pre&gt;&lt;code&gt;&amp;gt; db.greetings.remove({lang: &amp;quot;fr&amp;quot;})
1110WriteResult({ &amp;quot;nRemoved&amp;quot; : 1 })
1111&lt;/code&gt;&lt;/pre&gt;
1112&lt;h3 id=&quot;delete_a_collection&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#delete_a_collection&quot;&gt;¶&lt;/a&gt;Delete a collection&lt;/h3&gt;
1113&lt;p&gt;We never really used the &lt;code&gt;goodbyes&lt;/code&gt; collection. Can we get rid of that?&lt;/p&gt;
1114&lt;pre&gt;&lt;code&gt;&amp;gt; db.goodbyes.drop()
1115true
1116&lt;/code&gt;&lt;/pre&gt;
1117&lt;p&gt;Yes, it is &lt;code&gt;true&lt;/code&gt; that we can &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.collection.drop/index.html&quot;&gt;&lt;code&gt;drop&lt;/code&gt;&lt;/a&gt; it.&lt;/p&gt;
1118&lt;h3 id=&quot;delete_a_database&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#delete_a_database&quot;&gt;¶&lt;/a&gt;Delete a database&lt;/h3&gt;
1119&lt;p&gt;Now, I will be honest, I don’t really like our &lt;code&gt;greetings&lt;/code&gt; database either. It stinks. Let’s get rid of it as well:&lt;/p&gt;
1120&lt;pre&gt;&lt;code&gt;&amp;gt; db.dropDatabase()
1121{ &amp;quot;dropped&amp;quot; : &amp;quot;helloworld&amp;quot;, &amp;quot;ok&amp;quot; : 1 }
1122&lt;/code&gt;&lt;/pre&gt;
1123&lt;p&gt;Yeah, take that! The &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/db.dropDatabase/&quot;&gt;&lt;code&gt;dropDatabase&lt;/code&gt;&lt;/a&gt; can be used to drop databases.&lt;/p&gt;
1124&lt;h2 id=&quot;references&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#references&quot;&gt;¶&lt;/a&gt;References&lt;/h2&gt;
1125&lt;p&gt;The examples in this post are all fictional, and the methods that could be used where taken from Classmate’s post, and of course &lt;a href=&quot;https://docs.mongodb.com/manual/reference/method/&quot;&gt;Mongo’s documentation&lt;/a&gt;.&lt;/p&gt;
1126&lt;/main&gt;
1127&lt;/body&gt;
1128&lt;/html&gt;
1129 </content></entry><entry><title>Introduction to Hadoop and its MapReduce</title><id>dist/introduction-to-hadoop-and-its-mapreduce/index.html</id><updated>2020-04-02T22:00:00+00:00</updated><published>2020-03-31T22:00:00+00:00</published><summary>Hadoop is an open-source, free, Java-based programming framework that helps processing large datasets in a distributed environment and the problems that arise when trying to harness the knowledge from BigData, capable of running on thousands of nodes and dealing with petabytes of data. It is based on Google File System (GFS) and originated from the work on the Nutch open-source project on search engines.</summary><content type="html" src="dist/introduction-to-hadoop-and-its-mapreduce/index.html">&lt;!DOCTYPE html&gt;
1130&lt;html&gt;
1131&lt;head&gt;
1132&lt;meta charset=&quot;utf-8&quot; /&gt;
1133&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1134&lt;title&gt;Introduction to Hadoop and its MapReduce&lt;/title&gt;
1135&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1136&lt;/head&gt;
1137&lt;body&gt;
1138&lt;main&gt;
1139&lt;p&gt;Hadoop is an open-source, free, Java-based programming framework that helps processing large datasets in a distributed environment and the problems that arise when trying to harness the knowledge from BigData, capable of running on thousands of nodes and dealing with petabytes of data. It is based on Google File System (GFS) and originated from the work on the Nutch open-source project on search engines.&lt;/p&gt;
1140&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-04-01&lt;br&gt;
1141Modified 2020-04-03&lt;/div&gt;
1142&lt;p&gt;Hadoop also offers a distributed filesystem (HDFS) enabling for fast transfer among nodes, and a way to program with MapReduce.&lt;/p&gt;
1143&lt;p&gt;It aims to strive for the 4 V’s: Volume, Variety, Veracity and Velocity. For veracity, it is a secure environment that can be trusted.&lt;/p&gt;
1144&lt;h2 class=&quot;title&quot; id=&quot;milestones&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#milestones&quot;&gt;¶&lt;/a&gt;Milestones&lt;/h2&gt;
1145&lt;p&gt;The creators of Hadoop are Doug Cutting and Mike Cafarella, who just wanted to design a search engine, Nutch, and quickly found the problems of dealing with large amounts of data. They found their solution with the papers Google published.&lt;/p&gt;
1146&lt;p&gt;The name comes from the plush of Cutting’s child, a yellow elephant.&lt;/p&gt;
1147&lt;ul&gt;
1148&lt;li&gt;In July 2005, Nutch used GFS to perform MapReduce operations.&lt;/li&gt;
1149&lt;li&gt;In February 2006, Nutch started a Lucene subproject which led to Hadoop.&lt;/li&gt;
1150&lt;li&gt;In April 2007, Yahoo used Hadoop in a 1 000-node cluster.&lt;/li&gt;
1151&lt;li&gt;In January 2008, Apache took over and made Hadoop a top-level project.&lt;/li&gt;
1152&lt;li&gt;In July 2008, Apache tested a 4000-node cluster. The performance was the fastest compared to other technologies that year.&lt;/li&gt;
1153&lt;li&gt;In May 2009, Hadoop sorted a petabyte of data in 17 hours.&lt;/li&gt;
1154&lt;li&gt;In December 2011, Hadoop reached 1.0.&lt;/li&gt;
1155&lt;li&gt;In May 2012, Hadoop 2.0 was released with the addition of YARN (Yet Another Resource Navigator) on top of HDFS, splitting MapReduce and other processes into separate components, greatly improving the fault tolerance.&lt;/li&gt;
1156&lt;/ul&gt;
1157&lt;p&gt;From here onwards, many other alternatives have born, like Spark, Hive &amp;amp; Drill, Kafka, HBase, built around the Hadoop ecosystem.&lt;/p&gt;
1158&lt;p&gt;As of 2017, Amazon has clusters between 1 and 100 nodes, Yahoo has over 100 000 CPUs running Hadoop, AOL has clusters with 50 machines, and Facebook has a 320-machine (2 560 cores) and 1.3PB of raw storage.&lt;/p&gt;
1159&lt;h2 id=&quot;why_not_use_rdbms_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#why_not_use_rdbms_&quot;&gt;¶&lt;/a&gt;Why not use RDBMS?&lt;/h2&gt;
1160&lt;p&gt;Relational database management systems simply cannot scale horizontally, and vertical scaling will require very expensive servers. Similar to RDBMS, Hadoop has a notion of jobs (analogous to transactions), but without ACID or concurrency control. Hadoop supports any form of data (unstructured or semi-structured) in read-only mode, and failures are common but there’s a simple yet efficient fault tolerance.&lt;/p&gt;
1161&lt;p&gt;So what problems does Hadoop solve? It solves the way we should think about problems, and distributing them, which is key to do anything related with BigData nowadays. We start working with clusters of nodes, and coordinating the jobs between them. Hadoop’s API makes this really easy.&lt;/p&gt;
1162&lt;p&gt;Hadoop also takes very seriously the loss of data with replication, and if a node falls, they are moved to a different node.&lt;/p&gt;
1163&lt;h2 id=&quot;major_components&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#major_components&quot;&gt;¶&lt;/a&gt;Major components&lt;/h2&gt;
1164&lt;p&gt;The previously-mentioned HDFS runs on commodity machine, which are cost-friendly. It is very fault-tolerant and efficient enough to process huge amounts of data, because it splits large files into smaller chunks (or blocks) that can be more easily handled. Multiple nodes can work on multiple chunks at the same time.&lt;/p&gt;
1165&lt;p&gt;NameNode stores the metadata of the various datablocks (map of blocks) along with their location. It is the brain and the master in Hadoop’s master-slave architecture, also known as the namespace, and makes use of the DataNode.&lt;/p&gt;
1166&lt;p&gt;A secondary NameNode is a replica that can be used if the first NameNode dies, so that Hadoop doesn’t shutdown and can restart.&lt;/p&gt;
1167&lt;p&gt;DataNode stores the blocks of data, and are the slaves in the architecture. This data is split into one or more files. Their only job is to manage this access to the data. They are often distributed among racks to avoid data lose.&lt;/p&gt;
1168&lt;p&gt;JobTracker creates and schedules jobs from the clients for either map or reduce operations.&lt;/p&gt;
1169&lt;p&gt;TaskTracker runs MapReduce tasks assigned to the current data node.&lt;/p&gt;
1170&lt;p&gt;When clients need data, they first interact with the NameNode and replies with the location of the data in the correct DataNode. Client proceeds with interaction with the DataNode.&lt;/p&gt;
1171&lt;h2 id=&quot;mapreduce&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#mapreduce&quot;&gt;¶&lt;/a&gt;MapReduce&lt;/h2&gt;
1172&lt;p&gt;MapReduce, as the name implies, is split into two steps: the map and the reduce. The map stage is the «divide and conquer» strategy, while the reduce part is about combining and reducing the results.&lt;/p&gt;
1173&lt;p&gt;The mapper has to process the input data (normally a file or directory), commonly line-by-line, and produce one or more outputs. The reducer uses all the results from the mapper as its input to produce a new output file itself.&lt;/p&gt;
1174&lt;p&gt;&lt;img src=&quot;bitmap.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
1175&lt;p&gt;When reading the data, some may be junk that we can choose to ignore. If it is valid data, however, we label it with a particular type that can be useful for the upcoming process. Hadoop is responsible for splitting the data accross the many nodes available to execute this process in parallel.&lt;/p&gt;
1176&lt;p&gt;There is another part to MapReduce, known as the Shuffle-and-Sort. In this part, types or categories from one node get moved to a different node. This happens with all nodes, so that every node can work on a complete category. These categories are known as «keys», and allows Hadoop to scale linearly.&lt;/p&gt;
1177&lt;h2 id=&quot;references&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#references&quot;&gt;¶&lt;/a&gt;References&lt;/h2&gt;
1178&lt;ul&gt;
1179&lt;li&gt;&lt;a href=&quot;https://youtu.be/oT7kczq5A-0&quot;&gt;YouTube – Hadoop Tutorial For Beginners | What Is Hadoop? | Hadoop Tutorial | Hadoop Training | Simplilearn&lt;/a&gt;&lt;/li&gt;
1180&lt;li&gt;&lt;a href=&quot;https://youtu.be/bcjSe0xCHbE&quot;&gt;YouTube – Learn MapReduce with Playing Cards&lt;/a&gt;&lt;/li&gt;
1181&lt;li&gt;&lt;a href=&quot;https://youtu.be/j8ehT1_G5AY?list=PLi4tp-TF_qjM_ed4lIzn03w7OnEh0D8Xi&quot;&gt;YouTube – Video Post #2: Hadoop para torpes (I)-¿Qué es y para qué sirve?&lt;/a&gt;&lt;/li&gt;
1182&lt;li&gt;&lt;a href=&quot;https://youtu.be/NQ8mjVPCDvk?list=PLi4tp-TF_qjM_ed4lIzn03w7OnEh0D8Xi&quot;&gt;Video Post #3: Hadoop para torpes (II)-¿Cómo funciona? HDFS y MapReduce&lt;/a&gt;&lt;/li&gt;
1183&lt;li&gt;&lt;a href=&quot;https://hadoop.apache.org/old/releases.html&quot;&gt;Apache Hadoop Releases&lt;/a&gt;&lt;/li&gt;
1184&lt;li&gt;&lt;a href=&quot;https://youtu.be/20qWx2KYqYg?list=PLi4tp-TF_qjM_ed4lIzn03w7OnEh0D8Xi&quot;&gt;Video Post #4: Hadoop para torpes (III y fin)- Ecosistema y distribuciones&lt;/a&gt;&lt;/li&gt;
1185&lt;li&gt;&lt;a href=&quot;http://www.hadoopbook.com/&quot;&gt;Chapter 2 – Hadoop: The Definitive Guide, Fourth Edition&lt;/a&gt; (&lt;a href=&quot;http://grut-computing.com/HadoopBook.pdf&quot;&gt;pdf,&lt;/a&gt;&lt;a href=&quot;http://www.hadoopbook.com/code.html&quot;&gt;code&lt;/a&gt;)&lt;/li&gt;
1186&lt;/ul&gt;
1187&lt;/main&gt;
1188&lt;/body&gt;
1189&lt;/html&gt;
1190 </content></entry><entry><title>Google’s BigTable</title><id>dist/googles-bigtable/index.html</id><updated>2020-04-02T22:00:00+00:00</updated><published>2020-03-31T22:00:00+00:00</published><summary>Let’s talk about BigTable, and why it is what it is. But before we get into that, let’s see some important aspects anybody should consider when dealing with a lot of data (something BigTable does!).</summary><content type="html" src="dist/googles-bigtable/index.html">&lt;!DOCTYPE html&gt;
1191&lt;html&gt;
1192&lt;head&gt;
1193&lt;meta charset=&quot;utf-8&quot; /&gt;
1194&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1195&lt;title&gt;Google’s BigTable&lt;/title&gt;
1196&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1197&lt;/head&gt;
1198&lt;body&gt;
1199&lt;main&gt;
1200&lt;p&gt;Let’s talk about BigTable, and why it is what it is. But before we get into that, let’s see some important aspects anybody should consider when dealing with a lot of data (something BigTable does!).&lt;/p&gt;
1201&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-04-01&lt;br&gt;
1202Modified 2020-04-03&lt;/div&gt;
1203&lt;h2 class=&quot;title&quot; id=&quot;the_basics&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#the_basics&quot;&gt;¶&lt;/a&gt;The basics&lt;/h2&gt;
1204&lt;p&gt;Converting a text document into a different format is often a great way to greatly speed up scanning of it in the future. It allows for efficient searches.&lt;/p&gt;
1205&lt;p&gt;In addition, you generally want to store everything in a single, giant file. This will save a lot of time opening and closing files, because everything is in the same file! One proposal to make this happen is &lt;a href=&quot;https://trec.nist.gov/file_help.html&quot;&gt;Web TREC&lt;/a&gt; (see also the &lt;a href=&quot;https://en.wikipedia.org/wiki/Text_Retrieval_Conference&quot;&gt;Wikipedia page on TREC&lt;/a&gt;), which is basically HTML but every document is properly delimited from one another.&lt;/p&gt;
1206&lt;p&gt;Because we will have a lot of data, it’s often a good idea to compress it. Most text consists of the same words, over and over again. Classic compression techniques such as &lt;code&gt;DEFLATE&lt;/code&gt; or &lt;code&gt;LZW&lt;/code&gt; do an excellent job here.&lt;/p&gt;
1207&lt;h2 id=&quot;so_what_s_bigtable_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#so_what_s_bigtable_&quot;&gt;¶&lt;/a&gt;So what’s BigTable?&lt;/h2&gt;
1208&lt;p&gt;Okay, enough of an introduction to the basics on storing data. BigTable is what Google uses to store documents, and it’s a customized approach to save, search and update web pages.&lt;/p&gt;
1209&lt;p&gt;BigTable is is a distributed storage system for managing structured data, able to scale to petabytes of data across thousands of commodity servers, with wide applicability, scalability, high performance, and high availability.&lt;/p&gt;
1210&lt;p&gt;In a way, it’s kind of like databases and shares many implementation strategies with them, like parallel databases, or main-memory databases, but of course, with a different schema.&lt;/p&gt;
1211&lt;p&gt;It consists of a big table known as the «Root tablet», with pointers to many other «tablets» (or metadata in between). These are stored in a replicated filesystem accessible by all BigTable servers. Any change to a tablet gets logged (said log also gets stored in a replicated filesystem).&lt;/p&gt;
1212&lt;p&gt;If any of the tablets servers gets locked, a different one can take its place, read the log and deal with the problem.&lt;/p&gt;
1213&lt;p&gt;There’s no query language, transactions occur at row-level only. Every read or write in a row is atomic. Each row stores a single web page, and by combining the row and column keys along with a timestamp, it is possible to retrieve a single cell in the row. More formally, it’s a map that looks like this:&lt;/p&gt;
1214&lt;pre&gt;&lt;code&gt;fetch(row: string, column: string, time: int64) -&amp;gt; string
1215&lt;/code&gt;&lt;/pre&gt;
1216&lt;p&gt;A row may have as many columns as it needs, and these column groups are the same for everyone (but the columns themselves may vary), which is importan to reduce disk read time.&lt;/p&gt;
1217&lt;p&gt;Rows are split in different tablets based on the row keys, which simplifies determining an appropriated server for them. The keys can be up to 64KB big, although most commonly they range 10-100 bytes.&lt;/p&gt;
1218&lt;h2 id=&quot;conclusions&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#conclusions&quot;&gt;¶&lt;/a&gt;Conclusions&lt;/h2&gt;
1219&lt;p&gt;BigTable is Google’s way to deal with large amounts of data on many of their services, and the ideas behind it are not too complex to understand.&lt;/p&gt;
1220&lt;/main&gt;
1221&lt;/body&gt;
1222&lt;/html&gt;
1223 </content></entry><entry><title>A practical example with Hadoop</title><id>dist/a-practical-example-with-hadoop/index.html</id><updated>2020-04-02T22:00:00+00:00</updated><published>2020-03-31T22:00:00+00:00</published><summary>In our </summary><content type="html" src="dist/a-practical-example-with-hadoop/index.html">&lt;!DOCTYPE html&gt;
1224&lt;html&gt;
1225&lt;head&gt;
1226&lt;meta charset=&quot;utf-8&quot; /&gt;
1227&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1228&lt;title&gt;A practical example with Hadoop&lt;/title&gt;
1229&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1230&lt;/head&gt;
1231&lt;body&gt;
1232&lt;main&gt;
1233&lt;p&gt;In our &lt;a href=&quot;/blog/ribw/introduction-to-hadoop-and-its-mapreduce/&quot;&gt;previous Hadoop post&lt;/a&gt;, we learnt what it is, how it originated, and how it works, from a theoretical standpoint. Here we will instead focus on a more practical example with Hadoop.&lt;/p&gt;
1234&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-04-01&lt;br&gt;
1235Modified 2020-04-03&lt;/div&gt;
1236&lt;p&gt;This post will showcase my own implementation to implement a word counter for any plain text document that you want to analyze.&lt;/p&gt;
1237&lt;h2 class=&quot;title&quot; id=&quot;installation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#installation&quot;&gt;¶&lt;/a&gt;Installation&lt;/h2&gt;
1238&lt;p&gt;Before running any piece of software, its executable code must first be downloaded into our computers so that we can run it. Head over to &lt;a href=&quot;http://hadoop.apache.org/releases.html&quot;&gt;Apache Hadoop’s releases&lt;/a&gt; and download the &lt;a href=&quot;https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz&quot;&gt;latest binary version&lt;/a&gt; at the time of writing (3.2.1).&lt;/p&gt;
1239&lt;p&gt;We will be using the &lt;a href=&quot;https://linuxmint.com/&quot;&gt;Linux Mint&lt;/a&gt; distribution because I love its simplicity, although the process shown here should work just fine on any similar Linux distribution such as &lt;a href=&quot;https://ubuntu.com/&quot;&gt;Ubuntu&lt;/a&gt;.&lt;/p&gt;
1240&lt;p&gt;Once the archive download is complete, extract it with any tool of your choice (graphical or using the terminal) and execute it. Make sure you have a version of Java installed, such as &lt;a href=&quot;https://openjdk.java.net/&quot;&gt;OpenJDK&lt;/a&gt;.&lt;/p&gt;
1241&lt;p&gt;Here are all the three steps in the command line:&lt;/p&gt;
1242&lt;pre&gt;&lt;code&gt;wget https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
1243tar xf hadoop-3.2.1.tar.gz
1244hadoop-3.2.1/bin/hadoop version
1245&lt;/code&gt;&lt;/pre&gt;
1246&lt;h2 id=&quot;processing_data&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#processing_data&quot;&gt;¶&lt;/a&gt;Processing data&lt;/h2&gt;
1247&lt;p&gt;To take advantage of Hadoop, we have to design our code to work in the MapReduce model. Both the map and reduce phase work on key-value pairs as input and output, and both have a programmer-defined function.&lt;/p&gt;
1248&lt;p&gt;We will use Java, because it’s a dependency that we already have anyway, so might as well.&lt;/p&gt;
1249&lt;p&gt;Our map function needs to split each of the lines we receive as input into words, and we will also convert them to lowercase, thus preparing the data for later use (counting words). There won’t be bad records, so we don’t have to worry about that.&lt;/p&gt;
1250&lt;p&gt;Copy or reproduce the following code in a file called &lt;code&gt;WordCountMapper.java&lt;/code&gt;, using any text editor of your choice:&lt;/p&gt;
1251&lt;pre&gt;&lt;code&gt;import java.io.IOException;
1252
1253import org.apache.hadoop.io.IntWritable;
1254import org.apache.hadoop.io.LongWritable;
1255import org.apache.hadoop.io.Text;
1256import org.apache.hadoop.mapreduce.Mapper;
1257
1258public class WordCountMapper extends Mapper&amp;lt;LongWritable, Text, Text, IntWritable&amp;gt; {
1259    @Override
1260    public void map(LongWritable key, Text value, Context context)
1261            throws IOException, InterruptedException {
1262        for (String word : value.toString().split(&amp;quot;\\W&amp;quot;)) {
1263            context.write(new Text(word.toLowerCase()), new IntWritable(1));
1264        }
1265    }
1266}
1267&lt;/code&gt;&lt;/pre&gt;
1268&lt;p&gt;Now, let’s create the &lt;code&gt;WordCountReducer.java&lt;/code&gt; file. Its job is to reduce the data from multiple values into just one. We do that by summing all the values (our word count so far):&lt;/p&gt;
1269&lt;pre&gt;&lt;code&gt;import java.io.IOException;
1270import java.util.Iterator;
1271
1272import org.apache.hadoop.io.IntWritable;
1273import org.apache.hadoop.io.Text;
1274import org.apache.hadoop.mapreduce.Reducer;
1275
1276public class WordCountReducer extends Reducer&amp;lt;Text, IntWritable, Text, IntWritable&amp;gt; {
1277    @Override
1278    public void reduce(Text key, Iterable&amp;lt;IntWritable&amp;gt; values, Context context)
1279            throws IOException, InterruptedException {
1280        int count = 0;
1281        for (IntWritable value : values) {
1282            count += value.get();
1283        }
1284        context.write(key, new IntWritable(count));
1285    }
1286}
1287&lt;/code&gt;&lt;/pre&gt;
1288&lt;p&gt;Let’s just take a moment to appreciate how absolutely tiny this code is, and it’s Java! Hadoop’s API is really awesome and lets us write such concise code to achieve what we need.&lt;/p&gt;
1289&lt;p&gt;Last, let’s write the &lt;code&gt;main&lt;/code&gt; method, or else we won’t be able to run it. In our new file &lt;code&gt;WordCount.java&lt;/code&gt;:&lt;/p&gt;
1290&lt;pre&gt;&lt;code&gt;import org.apache.hadoop.fs.Path;
1291import org.apache.hadoop.io.IntWritable;
1292import org.apache.hadoop.io.Text;
1293import org.apache.hadoop.mapreduce.Job;
1294import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
1295import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
1296
1297public class WordCount {
1298    public static void main(String[] args) throws Exception {
1299        if (args.length != 2) {
1300            System.err.println(&amp;quot;usage: java WordCount &amp;lt;input path&amp;gt; &amp;lt;output path&amp;gt;&amp;quot;);
1301            System.exit(-1);
1302        }
1303
1304        Job job = Job.getInstance();
1305
1306        job.setJobName(&amp;quot;Word count&amp;quot;);
1307        job.setJarByClass(WordCount.class);
1308        job.setMapperClass(WordCountMapper.class);
1309        job.setReducerClass(WordCountReducer.class);
1310        job.setOutputKeyClass(Text.class);
1311        job.setOutputValueClass(IntWritable.class);
1312
1313        FileInputFormat.addInputPath(job, new Path(args[0]));
1314        FileOutputFormat.setOutputPath(job, new Path(args[1]));
1315
1316        boolean result = job.waitForCompletion(true);
1317
1318        System.exit(result ? 0 : 1);
1319    }
1320}
1321&lt;/code&gt;&lt;/pre&gt;
1322&lt;p&gt;And compile by including the required &lt;code&gt;.jar&lt;/code&gt; dependencies in Java’s classpath with the &lt;code&gt;-cp&lt;/code&gt; switch:&lt;/p&gt;
1323&lt;pre&gt;&lt;code&gt;javac -cp &amp;quot;hadoop-3.2.1/share/hadoop/common/*:hadoop-3.2.1/share/hadoop/mapreduce/*&amp;quot; *.java
1324&lt;/code&gt;&lt;/pre&gt;
1325&lt;p&gt;At last, we can run it (also specifying the dependencies in the classpath, this one’s a mouthful). Let’s run it on the same &lt;code&gt;WordCount.java&lt;/code&gt; source file we wrote:&lt;/p&gt;
1326&lt;pre&gt;&lt;code&gt;java -cp &amp;quot;.:hadoop-3.2.1/share/hadoop/common/*:hadoop-3.2.1/share/hadoop/common/lib/*:hadoop-3.2.1/share/hadoop/mapreduce/*:hadoop-3.2.1/share/hadoop/mapreduce/lib/*:hadoop-3.2.1/share/hadoop/yarn/*:hadoop-3.2.1/share/hadoop/yarn/lib/*:hadoop-3.2.1/share/hadoop/hdfs/*:hadoop-3.2.1/share/hadoop/hdfs/lib/*&amp;quot; WordCount WordCount.java results
1327&lt;/code&gt;&lt;/pre&gt;
1328&lt;p&gt;Hooray! We should have a new &lt;code&gt;results/&lt;/code&gt; folder along with the following files:&lt;/p&gt;
1329&lt;pre&gt;&lt;code&gt;$ ls results
1330part-r-00000  _SUCCESS
1331$ cat results/part-r-00000 
1332	154
13330	2
13341	3
13352	1
1336addinputpath	1
1337apache	6
1338args	4
1339boolean	1
1340class	6
1341count	1
1342err	1
1343exception	1
1344-snip- (output cut for clarity)
1345&lt;/code&gt;&lt;/pre&gt;
1346&lt;p&gt;It worked! Now this example was obviously tiny, but hopefully enough to demonstrate how to get the basics running on real world data.&lt;/p&gt;
1347&lt;/main&gt;
1348&lt;/body&gt;
1349&lt;/html&gt;
1350 </content></entry><entry><title>How does Google’s Search Engine work?</title><id>dist/how-does-googles-search-engine-work/index.html</id><updated>2020-03-27T23:00:00+00:00</updated><published>2020-03-17T23:00:00+00:00</published><summary>The original implementation was written in C/++ for Linux/Solaris.</summary><content type="html" src="dist/how-does-googles-search-engine-work/index.html">&lt;!DOCTYPE html&gt;
1351&lt;html&gt;
1352&lt;head&gt;
1353&lt;meta charset=&quot;utf-8&quot; /&gt;
1354&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1355&lt;title&gt;How does Google’s Search Engine work?&lt;/title&gt;
1356&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1357&lt;/head&gt;
1358&lt;body&gt;
1359&lt;main&gt;
1360&lt;p&gt;The original implementation was written in C/++ for Linux/Solaris.&lt;/p&gt;
1361&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-18&lt;br&gt;
1362Modified 2020-03-28&lt;/div&gt;
1363&lt;p&gt;There are three major components in the system’s anatomy, which can be thought as steps to be performed for Google to be what it is today.&lt;/p&gt;
1364&lt;p&gt;&lt;img src=&quot;image-1024x649.png&quot; alt=&quot;&quot; /&gt;&lt;/p&gt;
1365&lt;p&gt;But before we talk about the different components, let’s take a look at how they store all of this information.&lt;/p&gt;
1366&lt;h2 class=&quot;title&quot; id=&quot;data_structures&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#data_structures&quot;&gt;¶&lt;/a&gt;Data structures&lt;/h2&gt;
1367&lt;p&gt;A «BigFile» is a virtual file addressable by 64 bits.&lt;/p&gt;
1368&lt;p&gt;There exists a repository with the full HTML of every page compressed, along with a document identifier, length and URL.&lt;/p&gt;
1369&lt;table class=&quot;&quot;&gt;
1370 &lt;tbody&gt;
1371  &lt;tr&gt;
1372   &lt;td&gt;
1373    sync
1374   &lt;/td&gt;
1375   &lt;td&gt;
1376    length
1377   &lt;/td&gt;
1378   &lt;td&gt;
1379    compressed packet
1380   &lt;/td&gt;
1381  &lt;/tr&gt;
1382 &lt;/tbody&gt;
1383&lt;/table&gt;
1384&lt;p&gt;The Document Index has the document identifier, a pointer into the repository, a checksum and various other statistics.&lt;/p&gt;
1385&lt;table class=&quot;&quot;&gt;
1386 &lt;tbody&gt;
1387  &lt;tr&gt;
1388   &lt;td&gt;
1389    doc id
1390   &lt;/td&gt;
1391   &lt;td&gt;
1392    ecode
1393   &lt;/td&gt;
1394   &lt;td&gt;
1395    url len
1396   &lt;/td&gt;
1397   &lt;td&gt;
1398    page len
1399   &lt;/td&gt;
1400   &lt;td&gt;
1401    url
1402   &lt;/td&gt;
1403   &lt;td&gt;
1404    page
1405   &lt;/td&gt;
1406  &lt;/tr&gt;
1407 &lt;/tbody&gt;
1408&lt;/table&gt;
1409&lt;p&gt;A Lexicon stores the repository of words, implemented with a hashtable over pointers linking to the barrels (sorted linked lists) of the Inverted Index.&lt;/p&gt;
1410&lt;table class=&quot;&quot;&gt;
1411 &lt;tbody&gt;
1412  &lt;tr&gt;
1413   &lt;td&gt;
1414    word id
1415   &lt;/td&gt;
1416   &lt;td&gt;
1417    n docs
1418   &lt;/td&gt;
1419  &lt;/tr&gt;
1420  &lt;tr&gt;
1421   &lt;td&gt;
1422    word id
1423   &lt;/td&gt;
1424   &lt;td&gt;
1425    n docs
1426   &lt;/td&gt;
1427  &lt;/tr&gt;
1428 &lt;/tbody&gt;
1429&lt;/table&gt;
1430&lt;p&gt;The Hit Lists store occurences of a word in a document.&lt;/p&gt;
1431&lt;table class=&quot;&quot;&gt;
1432 &lt;tbody&gt;
1433  &lt;tr&gt;
1434   &lt;td&gt;
1435    &lt;strong&gt;
1436     plain
1437    &lt;/strong&gt;
1438   &lt;/td&gt;
1439   &lt;td&gt;
1440    cap: 1
1441   &lt;/td&gt;
1442   &lt;td&gt;
1443    imp: 3
1444   &lt;/td&gt;
1445   &lt;td&gt;
1446    pos: 12
1447   &lt;/td&gt;
1448  &lt;/tr&gt;
1449  &lt;tr&gt;
1450   &lt;td&gt;
1451    &lt;strong&gt;
1452     fancy
1453    &lt;/strong&gt;
1454   &lt;/td&gt;
1455   &lt;td&gt;
1456    cap: 1
1457   &lt;/td&gt;
1458   &lt;td&gt;
1459    imp: 7
1460   &lt;/td&gt;
1461   &lt;td&gt;
1462    type: 4
1463   &lt;/td&gt;
1464   &lt;td&gt;
1465    pos: 8
1466   &lt;/td&gt;
1467  &lt;/tr&gt;
1468  &lt;tr&gt;
1469   &lt;td&gt;
1470    &lt;strong&gt;
1471     anchor
1472    &lt;/strong&gt;
1473   &lt;/td&gt;
1474   &lt;td&gt;
1475    cap: 1
1476   &lt;/td&gt;
1477   &lt;td&gt;
1478    imp: 7
1479   &lt;/td&gt;
1480   &lt;td&gt;
1481    type: 4
1482   &lt;/td&gt;
1483   &lt;td&gt;
1484    hash: 4
1485   &lt;/td&gt;
1486   &lt;td&gt;
1487    pos: 8
1488   &lt;/td&gt;
1489  &lt;/tr&gt;
1490 &lt;/tbody&gt;
1491&lt;/table&gt;
1492&lt;p&gt;The Forward Index is a barrel with a range of word identifiers (document identifier and list of word identifiers).&lt;/p&gt;
1493&lt;table class=&quot;&quot;&gt;
1494 &lt;tbody&gt;
1495  &lt;tr&gt;
1496   &lt;td rowspan=&quot;3&quot;&gt;
1497    doc id
1498   &lt;/td&gt;
1499   &lt;td&gt;
1500    word id: 24
1501   &lt;/td&gt;
1502   &lt;td&gt;
1503    n hits: 8
1504   &lt;/td&gt;
1505   &lt;td&gt;
1506    hit hit hit hit hit hit hit hit
1507   &lt;/td&gt;
1508  &lt;/tr&gt;
1509  &lt;tr&gt;
1510   &lt;td&gt;
1511    word id: 24
1512   &lt;/td&gt;
1513   &lt;td&gt;
1514    n hits: 8
1515   &lt;/td&gt;
1516   &lt;td&gt;
1517    hit hit hit hit hit hit hit hit
1518   &lt;/td&gt;
1519  &lt;/tr&gt;
1520  &lt;tr&gt;
1521   &lt;td&gt;
1522    null word id
1523   &lt;/td&gt;
1524  &lt;/tr&gt;
1525 &lt;/tbody&gt;
1526&lt;/table&gt;
1527&lt;p&gt;The Inverted Index can be sorted by either document identifier or by ranking of word occurence.&lt;/p&gt;
1528&lt;table class=&quot;&quot;&gt;
1529 &lt;tbody&gt;
1530  &lt;tr&gt;
1531   &lt;td&gt;
1532    doc id: 23
1533   &lt;/td&gt;
1534   &lt;td&gt;
1535    n hits: 5
1536   &lt;/td&gt;
1537   &lt;td&gt;
1538    hit hit hit hit hit
1539   &lt;/td&gt;
1540  &lt;/tr&gt;
1541  &lt;tr&gt;
1542   &lt;td&gt;
1543    doc id: 23
1544   &lt;/td&gt;
1545   &lt;td&gt;
1546    n hits: 3
1547   &lt;/td&gt;
1548   &lt;td&gt;
1549    hit hit hit
1550   &lt;/td&gt;
1551  &lt;/tr&gt;
1552  &lt;tr&gt;
1553   &lt;td&gt;
1554    doc id: 23
1555   &lt;/td&gt;
1556   &lt;td&gt;
1557    n hits: 4
1558   &lt;/td&gt;
1559   &lt;td&gt;
1560    hit hit hit hit
1561   &lt;/td&gt;
1562  &lt;/tr&gt;
1563  &lt;tr&gt;
1564   &lt;td&gt;
1565    doc id: 23
1566   &lt;/td&gt;
1567   &lt;td&gt;
1568    n hits: 2
1569   &lt;/td&gt;
1570   &lt;td&gt;
1571    hit hit
1572   &lt;/td&gt;
1573  &lt;/tr&gt;
1574 &lt;/tbody&gt;
1575&lt;/table&gt;
1576&lt;p&gt;Back in 1998, Google compressed its repository to 53GB and had 24 million pages. The indices, lexicon, and other temporary storage required about 55GB.&lt;/p&gt;
1577&lt;h2 id=&quot;crawling&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#crawling&quot;&gt;¶&lt;/a&gt;Crawling&lt;/h2&gt;
1578&lt;p&gt;The crawling must be reliable, fast and robust, and also respect the decision of some authors not wanting their pages crawled. Originally, it took a week or more, so simultaneous execution became a must.&lt;/p&gt;
1579&lt;p&gt;Back in 1998, Google had between 3 and 4 crawlers running at 100 web pages per second maximum. These were implemented in Python.&lt;/p&gt;
1580&lt;p&gt;The crawled pages need parsing to deal with typos or formatting issues.&lt;/p&gt;
1581&lt;h2 id=&quot;indexing&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#indexing&quot;&gt;¶&lt;/a&gt;Indexing&lt;/h2&gt;
1582&lt;p&gt;Indexing is about putting the pages into barrels, converting words into word identifiers, and occurences into hit lists.&lt;/p&gt;
1583&lt;p&gt;Once indexing is done, sorting of the barrels happens to have them ordered by word identifier, producing the inverted index. This process also had to be done in parallel over many machines, or would otherwise have been too slow.&lt;/p&gt;
1584&lt;h2 id=&quot;searching&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#searching&quot;&gt;¶&lt;/a&gt;Searching&lt;/h2&gt;
1585&lt;p&gt;We need to find quality results efficiently. Plenty of weights are considered nowadays, but at its heart, PageRank is used. It is the algorithm they use to map the web, which is formally defined as follows:&lt;/p&gt;
1586&lt;p&gt;&lt;img src=&quot;8e1e61b119e107fcb4bdd7e78f649985.png&quot; alt=&quot;&quot; /&gt;
1587&lt;em&gt;PR(A) = (1-d) + d(PR(T1)/C(T1) + … + PR(Tn)/C(Tn))&lt;/em&gt;&lt;/p&gt;
1588&lt;p&gt;Where:&lt;/p&gt;
1589&lt;ul&gt;
1590&lt;li&gt;&lt;code&gt;A&lt;/code&gt; is a given page&lt;/li&gt;
1591&lt;li&gt;&lt;code&gt;T&amp;lt;sub&amp;gt;n&amp;lt;/sub&amp;gt;&lt;/code&gt; are pages that point to A&lt;/li&gt;
1592&lt;li&gt;&lt;code&gt;d&lt;/code&gt; is the damping factor in the range &lt;code&gt;[0, 1]&lt;/code&gt; (often 0.85)&lt;/li&gt;
1593&lt;li&gt;&lt;code&gt;C(A)&lt;/code&gt; is the number of links going out of page &lt;code&gt;A&lt;/code&gt;&lt;/li&gt;
1594&lt;li&gt;&lt;code&gt;PR(A)&lt;/code&gt; is the page rank of page &lt;code&gt;A&lt;/code&gt;
1595This formula indicates the probability that a random surfer visits a certain page, and &lt;code&gt;1 - d&lt;/code&gt; is used to indicate when it will «get bored» and stop surfing. More intuitively, the page rank of a page will grow as more pages link to it, or the few that link to it have high page rank.&lt;/li&gt;
1596&lt;/ul&gt;
1597&lt;p&gt;The anchor text in the links also help provide a better description and helps indexing for even better results.&lt;/p&gt;
1598&lt;p&gt;While searching, the concern is disk I/O which takes up most of the time. Caching is very important to improve performance up to 30 times.&lt;/p&gt;
1599&lt;p&gt;Now, in order to turn user queries into something we can search, we must parse the query and convert the words into word identifiers.&lt;/p&gt;
1600&lt;h2 id=&quot;conclusion&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#conclusion&quot;&gt;¶&lt;/a&gt;Conclusion&lt;/h2&gt;
1601&lt;p&gt;Google is designed to be a efficient, scalable, high-quality search engine. There are still bottlenecks in CPU, memory, disk speed and network I/O, but major data structures are used to make efficient use of the resources.&lt;/p&gt;
1602&lt;h2 id=&quot;references&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#references&quot;&gt;¶&lt;/a&gt;References&lt;/h2&gt;
1603&lt;ul&gt;
1604&lt;li&gt;&lt;a href=&quot;https://snap.stanford.edu/class/cs224w-readings/Brin98Anatomy.pdf&quot;&gt;The anatomy of a large-scale hypertextual Web search engine&lt;/a&gt;&lt;/li&gt;
1605&lt;li&gt;&lt;a href=&quot;https://www.site.uottawa.ca/%7Ediana/csi4107/Google_SearchEngine.pdf&quot;&gt;The Anatomy of a Large-Scale Hypertextual Web Search Engine (slides)&lt;/a&gt;&lt;/li&gt;
1606&lt;/ul&gt;
1607&lt;/main&gt;
1608&lt;/body&gt;
1609&lt;/html&gt;
1610 </content></entry><entry><title>Privado: PC-Crawler evaluation 2</title><id>dist/pc-crawler-evaluation-2/index.html</id><updated>2020-03-27T23:00:00+00:00</updated><published>2020-03-15T23:00:00+00:00</published><summary>As the student </summary><content type="html" src="dist/pc-crawler-evaluation-2/index.html">&lt;!DOCTYPE html&gt;
1611&lt;html&gt;
1612&lt;head&gt;
1613&lt;meta charset=&quot;utf-8&quot; /&gt;
1614&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1615&lt;title&gt;Privado: PC-Crawler evaluation 2&lt;/title&gt;
1616&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1617&lt;/head&gt;
1618&lt;body&gt;
1619&lt;main&gt;
1620&lt;p&gt;As the student &lt;code&gt;a(i)&lt;/code&gt; where &lt;code&gt;i = 9&lt;/code&gt;, I have been assigned to evaluate students &lt;code&gt;a(i - 1)&lt;/code&gt; and &lt;code&gt;a(i - 2)&lt;/code&gt;, these being:&lt;/p&gt;
1621&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-16&lt;br&gt;
1622Modified 2020-03-28&lt;/div&gt;
1623&lt;ul&gt;
1624&lt;li&gt;a08: Classmate (username)&lt;/li&gt;
1625&lt;li&gt;a07: Classmate (username)&lt;/li&gt;
1626&lt;/ul&gt;
1627&lt;p&gt;The evaluation is done according to the criteria described in Segunda entrega del PC-Crawler.&lt;/p&gt;
1628&lt;h2 class=&quot;title&quot; id=&quot;classmate_s_evaluation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#classmate_s_evaluation&quot;&gt;¶&lt;/a&gt;Classmate’s evaluation&lt;/h2&gt;
1629&lt;p&gt;&lt;strong&gt;Grading: A.&lt;/strong&gt;&lt;/p&gt;
1630&lt;p&gt;This is the evaluation of Crawler – Thesauro.&lt;/p&gt;
1631&lt;p&gt;It’s a well-written post, properly using WordPress code blocks, and they explain the process of improving the code and what it does. Because there are no noticeable issues with the post, they get the highest grading.&lt;/p&gt;
1632&lt;h2 id=&quot;classmate_s_evaluation_2&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#classmate_s_evaluation_2&quot;&gt;¶&lt;/a&gt;Classmate’s evaluation&lt;/h2&gt;
1633&lt;p&gt;&lt;strong&gt;Grading: B.&lt;/strong&gt;&lt;/p&gt;
1634&lt;p&gt;This is the evaluation of Actividad 2-Crawler.&lt;/p&gt;
1635&lt;p&gt;They start with an introduction on what they will do.&lt;/p&gt;
1636&lt;p&gt;Next, they show the code they have written, also describing what it does, although they don’t explain &lt;em&gt;why&lt;/em&gt; they chose the data structures they used.&lt;/p&gt;
1637&lt;p&gt;The style of the code leaves a lot to be desired, and they should have embedded the code in the post instead of taking screenshots. People that rely on screen readers will not be able to see the code.&lt;/p&gt;
1638&lt;p&gt;I have graded them B and not A for this last reason.&lt;/p&gt;
1639&lt;/main&gt;
1640&lt;/body&gt;
1641&lt;/html&gt;
1642 </content></entry><entry><title>What is ElasticSearch and why should you care?</title><id>dist/what-is-elasticsearch-and-why-should-you-care/index.html</id><updated>2020-03-26T23:00:00+00:00</updated><published>2020-03-17T23:00:00+00:00</published><summary>ElasticSearch is a giant search index with powerful analytics capabilities. It’s like a database and search engine on steroids, really easy and fast to get up and running. One can think of it as your own Google, a search engine with analytics.</summary><content type="html" src="dist/what-is-elasticsearch-and-why-should-you-care/index.html">&lt;!DOCTYPE html&gt;
1643&lt;html&gt;
1644&lt;head&gt;
1645&lt;meta charset=&quot;utf-8&quot; /&gt;
1646&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1647&lt;title&gt;What is ElasticSearch and why should you care?&lt;/title&gt;
1648&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1649&lt;/head&gt;
1650&lt;body&gt;
1651&lt;main&gt;
1652&lt;p&gt;ElasticSearch is a giant search index with powerful analytics capabilities. It’s like a database and search engine on steroids, really easy and fast to get up and running. One can think of it as your own Google, a search engine with analytics.&lt;/p&gt;
1653&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-18&lt;br&gt;
1654Modified 2020-03-27&lt;/div&gt;
1655&lt;p&gt;ElasticSearch is rich, stable, performs well, is well maintained, and able to scale to petabytes of any kind of data, whether it’s structured, semi-structured or not at all. It’s cost-effective and can be used to make business decisions.&lt;/p&gt;
1656&lt;p&gt;Or, described in 10 seconds:&lt;/p&gt;
1657&lt;blockquote&gt;
1658&lt;p&gt;Schema-free, REST &amp;amp; JSON based distributed document store
1659Open source: Apache License 2.0
1660Zero configuration&lt;/p&gt;
1661&lt;/blockquote&gt;
1662&lt;p&gt;-- Alex Reelsen&lt;/p&gt;
1663&lt;h2 class=&quot;title&quot; id=&quot;basic_capabilities&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#basic_capabilities&quot;&gt;¶&lt;/a&gt;Basic capabilities&lt;/h2&gt;
1664&lt;p&gt;ElasticSearch lets you ask questions about your data, not just make queries. You may think SQL can do this too, but what’s important is making a pipeline of facets, and feed the results from query to query.&lt;/p&gt;
1665&lt;p&gt;Instead of changing your data, you can be flexible with your questions with no need to re-index it every time the questions change.&lt;/p&gt;
1666&lt;p&gt;ElasticSearch is not just to search for full-text data, either. It can search for structured data and return more than just the results. It also yields additional data, such as ranking, highlights, and allows for pagination.&lt;/p&gt;
1667&lt;p&gt;It doesn’t take a lot of configuration to get running, either, which can be a good boost on productivity.&lt;/p&gt;
1668&lt;h2 id=&quot;how_does_it_work_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#how_does_it_work_&quot;&gt;¶&lt;/a&gt;How does it work?&lt;/h2&gt;
1669&lt;p&gt;ElasticSearch depends on Java, and can work in a distributed cluster if you execute multiple instances. Data will be replicated and sharded as needed. The current version at the time of writing is 7.6.1, and it’s being developed fast!&lt;/p&gt;
1670&lt;p&gt;It also has support for plugins, with an ever-growing ecosystem and integration on many programming languages. Tools around it are being built around it, too, like Kibana which helps you visualize your data.&lt;/p&gt;
1671&lt;p&gt;The way you use it is through a JSON API, served over HTTP/S.&lt;/p&gt;
1672&lt;h2 id=&quot;how_can_i_use_it_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#how_can_i_use_it_&quot;&gt;¶&lt;/a&gt;How can I use it?&lt;/h2&gt;
1673&lt;p&gt;&lt;a href=&quot;https://www.elastic.co/downloads/&quot;&gt;You can try ElasticSearch out for free on Elastic Cloud&lt;/a&gt;, however, it can also be &lt;a href=&quot;https://www.elastic.co/downloads/elasticsearch&quot;&gt;downloaded and ran offline&lt;/a&gt;, which is what we’ll do. Download the file corresponding to your operating system, unzip it, and execute the binary. Running it is as simple as that!&lt;/p&gt;
1674&lt;p&gt;Now you can make queries to it over HTTP, with for example &lt;code&gt;curl&lt;/code&gt;:&lt;/p&gt;
1675&lt;pre&gt;&lt;code&gt;curl -X PUT localhost:9200/orders/order/1 -d '
1676{
1677  &amp;quot;created_at&amp;quot;: &amp;quot;2013/09/05 15:45:10&amp;quot;,
1678  &amp;quot;items&amp;quot;: [
1679    {
1680      name: &amp;quot;HD Monitor&amp;quot;
1681    }
1682  ],
1683  &amp;quot;total&amp;quot;: 249.95
1684}'
1685&lt;/code&gt;&lt;/pre&gt;
1686&lt;p&gt;This will create a new order with some information, such as when it was created, what items it contains, and the total cost of the order.&lt;/p&gt;
1687&lt;p&gt;You can then query or filter as needed, script it or even create statistics.&lt;/p&gt;
1688&lt;h2 id=&quot;references&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#references&quot;&gt;¶&lt;/a&gt;References&lt;/h2&gt;
1689&lt;ul&gt;
1690&lt;li&gt;&lt;a href=&quot;https://youtu.be/sKnkQSec1U0&quot;&gt;YouTube – What is Elasticsearch?&lt;/a&gt;&lt;/li&gt;
1691&lt;li&gt;&lt;a href=&quot;https://youtu.be/yWNiRC_hUAw&quot;&gt;YouTube – GOTO 2013 • Elasticsearch – Beyond Full-text Search • Alex Reelsen&lt;/a&gt;&lt;/li&gt;
1692&lt;li&gt;&lt;a href=&quot;https://www.elastic.co/kibana&quot;&gt;Kibana – Your window into the Elastic Stack&lt;/a&gt;&lt;/li&gt;
1693&lt;li&gt;&lt;a href=&quot;https://www.elastic.co/guide/index.html&quot;&gt;Elastic Stack and Product Documentation&lt;/a&gt;&lt;/li&gt;
1694&lt;/ul&gt;
1695&lt;/main&gt;
1696&lt;/body&gt;
1697&lt;/html&gt;
1698 </content></entry><entry><title>Privado: NoSQL evaluation</title><id>dist/nosql-evaluation/index.html</id><updated>2020-03-26T23:00:00+00:00</updated><published>2020-03-15T23:00:00+00:00</published><summary>I have decided to evaluate Classmate‘s post and Classmate‘s post, because they review databases I have not seen or used before, and I think it would be interesting to see new ones.</summary><content type="html" src="dist/nosql-evaluation/index.html">&lt;!DOCTYPE html&gt;
1699&lt;html&gt;
1700&lt;head&gt;
1701&lt;meta charset=&quot;utf-8&quot; /&gt;
1702&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1703&lt;title&gt;Privado: NoSQL evaluation&lt;/title&gt;
1704&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1705&lt;/head&gt;
1706&lt;body&gt;
1707&lt;main&gt;
1708&lt;p&gt;I have decided to evaluate Classmate‘s post and Classmate‘s post, because they review databases I have not seen or used before, and I think it would be interesting to see new ones.&lt;/p&gt;
1709&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-16&lt;br&gt;
1710Modified 2020-03-27&lt;/div&gt;
1711&lt;p&gt;The evaluation is based on the requirements defined by Trabajos en grupo sobre Bases de Datos NoSQL:&lt;/p&gt;
1712&lt;blockquote&gt;
1713&lt;p&gt;&lt;strong&gt;1ª entrada:&lt;/strong&gt; Descripción de la finalidad de la tecnología y cómo funciona o trabaja la BD NoSQL, sus características, la arista que ocupa en el Teorema CAP, de dónde se descarga, y cómo se instala.&lt;/p&gt;
1714&lt;/blockquote&gt;
1715&lt;p&gt;-- Teacher&lt;/p&gt;
1716&lt;h2 class=&quot;title&quot; id=&quot;classmate_s_evaluation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#classmate_s_evaluation&quot;&gt;¶&lt;/a&gt;Classmate’s evaluation&lt;/h2&gt;
1717&lt;p&gt;&lt;strong&gt;Grading: A.&lt;/strong&gt;&lt;/p&gt;
1718&lt;p&gt;The post I have evaluated is BB.DD. NoSQL: Voldemort 1ª Fase.&lt;/p&gt;
1719&lt;p&gt;The post doesn’t start very well, because the first sentence has (emphasis mine):&lt;/p&gt;
1720&lt;blockquote&gt;
1721&lt;p&gt;En él repasaremos en qué consiste &lt;strong&gt;MongoDB&lt;/strong&gt;, sus características, y cómo se instala, entre otros.&lt;/p&gt;
1722&lt;/blockquote&gt;
1723&lt;p&gt;-- Classmate&lt;/p&gt;
1724&lt;p&gt;…yet the post is about Voldemort!&lt;/p&gt;
1725&lt;p&gt;The post does detail how it works, its architecture, corner in the CAP theorem, download and installation.&lt;/p&gt;
1726&lt;p&gt;I have graded the post with A because I think it meets all the requirements, even if they slipped a bit in the beginning.&lt;/p&gt;
1727&lt;h2 id=&quot;classmate_s_evaluation_2&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#classmate_s_evaluation_2&quot;&gt;¶&lt;/a&gt;Classmate’s evaluation&lt;/h2&gt;
1728&lt;p&gt;&lt;strong&gt;Grading: A.&lt;/strong&gt;&lt;/p&gt;
1729&lt;p&gt;The post I have evaluted is Raven.&lt;/p&gt;
1730&lt;p&gt;They have done a good job describing the project’s goals, corner in the CAP theorem, download, and provide an extensive installation section.&lt;/p&gt;
1731&lt;p&gt;They don’t seem to use some of WordPress features, such as lists, but otherwise the post is good and deserves an A grading.&lt;/p&gt;
1732&lt;/main&gt;
1733&lt;/body&gt;
1734&lt;/html&gt;
1735 </content></entry><entry><title>Integrating Apache Tika into our Crawler</title><id>dist/integrating-apache-tika-into-our-crawler/index.html</id><updated>2020-03-24T23:00:00+00:00</updated><published>2020-03-17T23:00:00+00:00</published><summary>In our last crawler post</summary><content type="html" src="dist/integrating-apache-tika-into-our-crawler/index.html">&lt;!DOCTYPE html&gt;
1736&lt;html&gt;
1737&lt;head&gt;
1738&lt;meta charset=&quot;utf-8&quot; /&gt;
1739&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1740&lt;title&gt;Integrating Apache Tika into our Crawler&lt;/title&gt;
1741&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1742&lt;/head&gt;
1743&lt;body&gt;
1744&lt;main&gt;
1745&lt;p&gt;&lt;a href=&quot;/blog/ribw/upgrading-our-baby-crawler/&quot;&gt;In our last crawler post&lt;/a&gt;, we detailed how our crawler worked, and although it did a fine job, it’s time for some extra upgrading.&lt;/p&gt;
1746&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-18&lt;br&gt;
1747Modified 2020-03-25&lt;/div&gt;
1748&lt;h2 class=&quot;title&quot; id=&quot;what_kind_of_upgrades_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#what_kind_of_upgrades_&quot;&gt;¶&lt;/a&gt;What kind of upgrades?&lt;/h2&gt;
1749&lt;p&gt;A small but useful one. We are adding support for file types that contain text but cannot be processed by normal text editors because they are structured and not just plain text (such as PDF files, Excel, Word documents…).&lt;/p&gt;
1750&lt;p&gt;And for this task, we will make use of the help offered by &lt;a href=&quot;https://tika.apache.org/&quot;&gt;Tika&lt;/a&gt;, our friendly Apache tool.&lt;/p&gt;
1751&lt;h2 id=&quot;what_is_tika_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#what_is_tika_&quot;&gt;¶&lt;/a&gt;What is Tika?&lt;/h2&gt;
1752&lt;p&gt;&lt;a href=&quot;https://tika.apache.org/&quot;&gt;Tika&lt;/a&gt; is a set of libraries offered by &lt;a href=&quot;https://en.wikipedia.org/wiki/The_Apache_Software_Foundation&quot;&gt;The Apache Software Foundation&lt;/a&gt; that we can include in our project in order to extract the text and metadata of files from a &lt;a href=&quot;https://tika.apache.org/1.24/formats.html&quot;&gt;long list of supported formats&lt;/a&gt;.&lt;/p&gt;
1753&lt;h2 id=&quot;changes_in_the_code&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#changes_in_the_code&quot;&gt;¶&lt;/a&gt;Changes in the code&lt;/h2&gt;
1754&lt;p&gt;Not much has changed in the structure of the crawler, we simply have added a new method in &lt;code&gt;Utils&lt;/code&gt; that uses the class &lt;code&gt;Tika&lt;/code&gt; from the previously mentioned library so as to process and extract the text of more filetypes.&lt;/p&gt;
1755&lt;p&gt;Then, we use this text just like we would for our standard text file (checking the thesaurus and adding it to the word map) and voilà! We have just added support for a big range of file types.&lt;/p&gt;
1756&lt;h2 id=&quot;incorporating_gradle&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#incorporating_gradle&quot;&gt;¶&lt;/a&gt;Incorporating Gradle&lt;/h2&gt;
1757&lt;p&gt;In order for the previous code to work, we need to make use of external libraries. To make this process easier and because the project is growing, we decided to use &lt;a href=&quot;https://gradle.org/&quot;&gt;Gradle&lt;/a&gt;, a build system that can be used for projects in various programming languages, such as Java.&lt;/p&gt;
1758&lt;p&gt;We followed their &lt;a href=&quot;https://guides.gradle.org/building-java-applications/&quot;&gt;guide to Building Java Applications&lt;/a&gt;, and in a few steps added the required &lt;code&gt;.gradle&lt;/code&gt; files. Now we can compile and run the code without having to worry about juggling with Java and external dependencies in a single command:&lt;/p&gt;
1759&lt;pre&gt;&lt;code&gt;./gradlew run
1760&lt;/code&gt;&lt;/pre&gt;
1761&lt;h2 id=&quot;download&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#download&quot;&gt;¶&lt;/a&gt;Download&lt;/h2&gt;
1762&lt;p&gt;And here you can download the final result:&lt;/p&gt;
1763&lt;p&gt;&lt;em&gt;download removed&lt;/em&gt;&lt;/p&gt;
1764&lt;/main&gt;
1765&lt;/body&gt;
1766&lt;/html&gt;
1767 </content></entry><entry><title>Cassandra: Basic Operations and Architecture</title><id>dist/nosql-databases-basic-operations-and-architecture/index.html</id><updated>2020-03-23T23:00:00+00:00</updated><published>2020-03-04T23:00:00+00:00</published><summary>This is the second post in the NoSQL Databases series, with a brief description on the basic operations (such as insertion, retrieval, indexing…), and complete execution along with the data model / architecture.</summary><content type="html" src="dist/nosql-databases-basic-operations-and-architecture/index.html">&lt;!DOCTYPE html&gt;
1768&lt;html&gt;
1769&lt;head&gt;
1770&lt;meta charset=&quot;utf-8&quot; /&gt;
1771&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
1772&lt;title&gt;Cassandra: Basic Operations and Architecture&lt;/title&gt;
1773&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
1774&lt;/head&gt;
1775&lt;body&gt;
1776&lt;main&gt;
1777&lt;p&gt;This is the second post in the NoSQL Databases series, with a brief description on the basic operations (such as insertion, retrieval, indexing…), and complete execution along with the data model / architecture.&lt;/p&gt;
1778&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-05&lt;br&gt;
1779Modified 2020-03-24&lt;/div&gt;
1780&lt;p&gt;Other posts in this series:&lt;/p&gt;
1781&lt;ul&gt;
1782&lt;li&gt;&lt;a href=&quot;/blog/ribw/nosql-databases-an-introduction/&quot;&gt;Cassandra: an Introduction&lt;/a&gt;&lt;/li&gt;
1783&lt;li&gt;&lt;a href=&quot;/blog/ribw/nosql-databases-basic-operations-and-architecture/&quot;&gt;Cassandra: Basic Operations and Architecture&lt;/a&gt; (this post)&lt;/li&gt;
1784&lt;/ul&gt;
1785&lt;hr /&gt;
1786&lt;p&gt;Cassandra uses it own Query Language for managing the databases, it is known as **CQL **(&lt;strong&gt;Cassandra Query Language&lt;/strong&gt;). Cassandra stores data in &lt;strong&gt;&lt;em&gt;tables&lt;/em&gt;&lt;/strong&gt;, as in relational databases, and these tables are grouped in &lt;strong&gt;&lt;em&gt;keyspaces&lt;/em&gt;&lt;/strong&gt;. A keyspace defines a number of options that applies to all the tables it contains. The most used option is the **replication strategy. **It is recommended to have only one keyspace by application.&lt;/p&gt;
1787&lt;p&gt;It is important to mention that &lt;strong&gt;tables and keyspaces&lt;/strong&gt; are** case insensitive**, so myTable is equivalent to mytable, but it is possible to &lt;strong&gt;force case sensitivity&lt;/strong&gt; using &lt;strong&gt;double-quotes&lt;/strong&gt;.&lt;/p&gt;
1788&lt;p&gt;To begin with the basic operations it is necessary to deploy Cassandra:&lt;/p&gt;
1789&lt;ol&gt;
1790&lt;li&gt;Open a terminal in the root of the Apache Cassandra folder downloaded in the previous post.&lt;/li&gt;
1791&lt;li&gt;Run the command:&lt;/li&gt;
1792&lt;/ol&gt;
1793&lt;pre&gt;&lt;code&gt;$ bin/cassandra
1794&lt;/code&gt;&lt;/pre&gt;
1795&lt;p&gt;Once Cassandra is deployed, it is time to open a** CQL Shell**, in &lt;strong&gt;other terminal&lt;/strong&gt;, with the command: &lt;/p&gt;
1796&lt;pre&gt;&lt;code&gt;$ bin/cqlsh
1797&lt;/code&gt;&lt;/pre&gt;
1798&lt;p&gt;It is possible to check if Cassandra is deployed if the SQL Shell prints the next message:&lt;/p&gt;
1799&lt;p&gt;&lt;img src=&quot;uwqQgQte-cuYb_pePFOuY58re23kngrDKNgL1qz4yOfnBDZkqMIH3fFuCrye.png&quot; alt=&quot;&quot; /&gt;
1800&lt;em&gt;CQL Shell&lt;/em&gt;&lt;/p&gt;
1801&lt;h2 class=&quot;title&quot; id=&quot;create_insert&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#create_insert&quot;&gt;¶&lt;/a&gt;Create/Insert&lt;/h2&gt;
1802&lt;h3 id=&quot;ddl_data_definition_language_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#ddl_data_definition_language_&quot;&gt;¶&lt;/a&gt;DDL (Data Definition Language)&lt;/h3&gt;
1803&lt;h4 id=&quot;create_keyspace&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#create_keyspace&quot;&gt;¶&lt;/a&gt;Create keyspace&lt;/h4&gt;
1804&lt;p&gt;A keyspace is created using a **CREATE KEYSPACE **statement:&lt;/p&gt;
1805&lt;pre&gt;&lt;code&gt;$ **CREATE** KEYSPACE [ **IF** **NOT** **EXISTS** ] keyspace_name **WITH** options;
1806&lt;/code&gt;&lt;/pre&gt;
1807&lt;p&gt;The supported “&lt;strong&gt;options&lt;/strong&gt;” are:&lt;/p&gt;
1808&lt;ul&gt;
1809&lt;li&gt;“&lt;strong&gt;replication&lt;/strong&gt;”: this is **mandatory **and defines the &lt;strong&gt;replication strategy&lt;/strong&gt; and the &lt;strong&gt;replication factor&lt;/strong&gt; (the number of nodes that will have a copy of the data). Within this option there is a property called “&lt;strong&gt;class&lt;/strong&gt;” in which the &lt;strong&gt;replication strategy&lt;/strong&gt; is specified (“SimpleStrategy” or “NetworkTopologyStrategy”)&lt;/li&gt;
1810&lt;li&gt;“&lt;strong&gt;durable_writes&lt;/strong&gt;”: this is &lt;strong&gt;not mandatory&lt;/strong&gt; and it is possible to use the &lt;strong&gt;commit logs for updates&lt;/strong&gt;.
1811Attempting to create an already existing keyspace will return an error unless the **IF NOT EXISTS **directive is used. &lt;/li&gt;
1812&lt;/ul&gt;
1813&lt;p&gt;The example associated to this statement is create a keyspace with name “test_keyspace” with “SimpleStrategy” as “class” of replication and a “replication_factor” of 3.&lt;/p&gt;
1814&lt;pre&gt;&lt;code&gt;**CREATE** KEYSPACE test_keyspace
1815    **WITH** **replication** = {'class': 'SimpleStrategy',
1816                        'replication_factor' : 3};
1817&lt;/code&gt;&lt;/pre&gt;
1818&lt;p&gt;The **USE **statement allows to &lt;strong&gt;change&lt;/strong&gt; the current &lt;strong&gt;keyspace&lt;/strong&gt;. The syntax of this statement is very simple: &lt;/p&gt;
1819&lt;pre&gt;&lt;code&gt;**USE** keyspace_name;
1820&lt;/code&gt;&lt;/pre&gt;
1821&lt;p&gt;&lt;img src=&quot;RDWIG2RwvEevUFQv6TGFtGzRm4_9ERpxPf0feriflaj3alvWw3FEIAr_ZdF1.png&quot; alt=&quot;&quot; /&gt;
1822&lt;em&gt;USE statement&lt;/em&gt;&lt;/p&gt;
1823&lt;p&gt;It is also possible to get the metadata from a keyspace with the **DESCRIBE **statement.&lt;/p&gt;
1824&lt;pre&gt;&lt;code&gt;**DESCRIBE** KEYSPACES | KEYSPACE keyspace_name;
1825&lt;/code&gt;&lt;/pre&gt;
1826&lt;h4 id=&quot;create_table&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#create_table&quot;&gt;¶&lt;/a&gt;Create table&lt;/h4&gt;
1827&lt;p&gt;Creating a new table uses the **CREATE TABLE **statement:&lt;/p&gt;
1828&lt;pre&gt;&lt;code&gt;**CREATE** **TABLE** [ **IF** **NOT** **EXISTS** ] table_name
1829    '('
1830        column_definition
1831        ( ',' column_definition )*
1832        [ ',' **PRIMARY** **KEY** '(' primary_key ')' ]
1833    ')' [ **WITH** table_options ];
1834&lt;/code&gt;&lt;/pre&gt;
1835&lt;p&gt;With “column_definition” as: column_name cql_type [ STATIC ] [ PRIMARY KEY]; “primary_key” as: partition_key [ ‘,’ clustering_columns ]; and “table_options” as: COMPACT STORAGE [ AND table_options ] or CLUSTERING ORDER BY ‘(‘ clustering_order ‘)’ [ AND table_options ] or “options”.&lt;/p&gt;
1836&lt;p&gt;Attempting to create an already existing table will return an error unless the &lt;strong&gt;IF NOT EXISTS&lt;/strong&gt; directive is used.&lt;/p&gt;
1837&lt;p&gt;The &lt;strong&gt;CQL types&lt;/strong&gt; are described in the References section.&lt;/p&gt;
1838&lt;p&gt;For example, we are going to create a table called “species_table” in the keyspace “test_keyspace” in which we will have a “species” text (as PRIMARY KEY), a “common_name” text, a “population” varint, a “average_size” int and a “sex” text. Besides, we are going to add a comment to the table: “Some species records”;&lt;/p&gt;
1839&lt;pre&gt;&lt;code&gt;**CREATE** **TABLE** species_table (
1840    species text **PRIMARY** **KEY**,
1841    common_name text,
1842    population varint,
1843    average_size **int**,
1844    sex text,
1845) **WITH** **comment**='Some species records';
1846&lt;/code&gt;&lt;/pre&gt;
1847&lt;p&gt;It is also possible to get the metadata from a table with the **DESCRIBE **statement.&lt;/p&gt;
1848&lt;pre&gt;&lt;code&gt;**DESCRIBE** **TABLES** | **TABLE** [keyspace_name.]table_name;
1849&lt;/code&gt;&lt;/pre&gt;
1850&lt;h3 id=&quot;dml_data_manipulation_language_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#dml_data_manipulation_language_&quot;&gt;¶&lt;/a&gt;DML (Data Manipulation Language)&lt;/h3&gt;
1851&lt;h4 id=&quot;insert_data&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#insert_data&quot;&gt;¶&lt;/a&gt;Insert data&lt;/h4&gt;
1852&lt;p&gt;Inserting data for a row is done using an **INSERT **statement:&lt;/p&gt;
1853&lt;pre&gt;&lt;code&gt;**INSERT** **INTO** table_name ( names_values | json_clause )
1854                      [ **IF** **NOT** **EXISTS** ]
1855                      [ **USING** update_parameter ( **AND** update_parameter )* ];
1856&lt;/code&gt;&lt;/pre&gt;
1857&lt;p&gt;Where “names_values” is: names VALUES tuple_literal; “json_clause” is: JSON string [ DEFAULT ( NULL | UNSET ) ]; and “update_parameter” is usually: TTL.&lt;/p&gt;
1858&lt;p&gt;For example we are going to use both VALUES and JSON clauses to insert data in the table “species_table”. In the VALUES clause it is necessary to supply the list of columns, not as in the JSON clause that is optional.&lt;/p&gt;
1859&lt;p&gt;Note: TTL (Time To Live) and Timestamp are metrics for expiring data, so, when the time set is passed, the operation is expired.&lt;/p&gt;
1860&lt;p&gt;In the VALUES clause we are going to insert a new specie called “White monkey”, with an average size of 3, its common name is “Copito de nieve”, population 0 and sex “male”.&lt;/p&gt;
1861&lt;pre&gt;&lt;code&gt;**INSERT** **INTO** species_table (species, common_name, population, average_size, sex)
1862                **VALUES** ('White monkey', 'Copito de nieve', 0, 3, 'male');
1863&lt;/code&gt;&lt;/pre&gt;
1864&lt;p&gt;In the JSON clause we are going to insert a new specie called “Cloned sheep”, with an average size of 1, its common name is “Dolly the sheep”, population 0 and sex “female”.&lt;/p&gt;
1865&lt;pre&gt;&lt;code&gt;**INSERT** **INTO** species_table JSON '{&amp;quot;species&amp;quot;: &amp;quot;Cloned Sheep&amp;quot;,
1866                              &amp;quot;common_name&amp;quot;: &amp;quot;Dolly the Sheep&amp;quot;,
1867                              &amp;quot;average_size&amp;quot;:1,
1868                              &amp;quot;population&amp;quot;:0,
1869                              &amp;quot;sex&amp;quot;: &amp;quot;female&amp;quot;}';
1870&lt;/code&gt;&lt;/pre&gt;
1871&lt;p&gt;Note: all updates for an **INSERT **are applied **atomically **and in &lt;strong&gt;isolation.&lt;/strong&gt;&lt;/p&gt;
1872&lt;h2 id=&quot;read&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#read&quot;&gt;¶&lt;/a&gt;Read&lt;/h2&gt;
1873&lt;p&gt;Querying data from data is done using a **SELECT **statement:&lt;/p&gt;
1874&lt;pre&gt;&lt;code&gt;**SELECT** [ JSON | **DISTINCT** ] ( select_clause | '*' )
1875                      **FROM** table_name
1876                      [ **WHERE** where_clause ]
1877                      [ **GROUP** **BY** group_by_clause ]
1878                      [ **ORDER** **BY** ordering_clause ]
1879                      [ PER **PARTITION** **LIMIT** (**integer** | bind_marker) ]
1880                      [ **LIMIT** (**integer** | bind_marker) ]
1881                      [ ALLOW FILTERING ];
1882&lt;/code&gt;&lt;/pre&gt;
1883&lt;p&gt;The **CQL SELECT **statement is very **similar **to the **SQL SELECT **statement due to the fact that both allows filtering (&lt;strong&gt;WHERE&lt;/strong&gt;), grouping data (&lt;strong&gt;GROUP BY&lt;/strong&gt;), ordering the data (&lt;strong&gt;ORDER BY&lt;/strong&gt;) and limit the number of data (&lt;strong&gt;LIMIT&lt;/strong&gt;). Besides, **CQL offers **a **limit per partition **and allow the **filtering **of &lt;strong&gt;data&lt;/strong&gt;.&lt;/p&gt;
1884&lt;p&gt;Note: as in SQL it it possible to set alias to the data with the statement &lt;strong&gt;AS.&lt;/strong&gt;&lt;/p&gt;
1885&lt;p&gt;For example we are going to retrieve all the information about those values from the tables “species_table” which “sex” is “male”. Allow filtering is mandatory when there is a WHERE statement.&lt;/p&gt;
1886&lt;pre&gt;&lt;code&gt;**SELECT** * **FROM** species_table **WHERE** sex = 'male' ALLOW FILTERING;
1887&lt;/code&gt;&lt;/pre&gt;
1888&lt;p&gt;&lt;img src=&quot;s6GrKIGATvOSD7oGRNScUU5RnLN_-3X1JXvnVi_wDT_hrmPMZdnCdBI8DpIJ.png&quot; alt=&quot;&quot; /&gt;
1889&lt;em&gt;SELECT statement&lt;/em&gt;&lt;/p&gt;
1890&lt;p&gt;Furthermore, we are going to test the SELECT JSON statement. For this, we are going to retrieve only the species name with a population of 0. &lt;/p&gt;
1891&lt;pre&gt;&lt;code&gt;**SELECT** JSON species **FROM** species_table **WHERE** population = 0 ALLOW FILTERING;
1892&lt;/code&gt;&lt;/pre&gt;
1893&lt;p&gt;&lt;img src=&quot;Up_eHlqKQp2RI5XIbgPOvj1B5J3gLxz7v7EI0NDRgezQTipecdfDT6AQoso0.png&quot; alt=&quot;&quot; /&gt;
1894&lt;em&gt;SELECT JSON statement&lt;/em&gt;&lt;/p&gt;
1895&lt;h2 id=&quot;update&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#update&quot;&gt;¶&lt;/a&gt;Update&lt;/h2&gt;
1896&lt;h3 id=&quot;ddl_data_definition_language__2&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#ddl_data_definition_language__2&quot;&gt;¶&lt;/a&gt;DDL (Data Definition Language)&lt;/h3&gt;
1897&lt;h4 id=&quot;alter_keyspace&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#alter_keyspace&quot;&gt;¶&lt;/a&gt;Alter keyspace&lt;/h4&gt;
1898&lt;p&gt;The statement **ALTER KEYSPACE **allows to modify the options of a keyspace:&lt;/p&gt;
1899&lt;pre&gt;&lt;code&gt;**ALTER** KEYSPACE keyspace_name **WITH** options;
1900&lt;/code&gt;&lt;/pre&gt;
1901&lt;p&gt;Note: the supported **options **are the same than for creating a keyspace, “&lt;strong&gt;replication&lt;/strong&gt;” and “&lt;strong&gt;durable_writes&lt;/strong&gt;”.&lt;/p&gt;
1902&lt;p&gt;The example associated to this statement is to modify the keyspace with name “test_keyspace” and set a “replication_factor” of 4.&lt;/p&gt;
1903&lt;pre&gt;&lt;code&gt;**ALTER** KEYSPACE test_keyspace
1904    **WITH** **replication** = {'class': 'SimpleStrategy', 'replication_factor' : 4};
1905&lt;/code&gt;&lt;/pre&gt;
1906&lt;h4 id=&quot;alter_table&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#alter_table&quot;&gt;¶&lt;/a&gt;Alter table&lt;/h4&gt;
1907&lt;p&gt;Altering an existing table uses the **ALTER TABLE **statement:&lt;/p&gt;
1908&lt;pre&gt;&lt;code&gt;**ALTER** **TABLE** table_name alter_table_instruction;
1909&lt;/code&gt;&lt;/pre&gt;
1910&lt;p&gt;Where “alter_table_instruction” can be: ADD column_name cql_type ( ‘,’ column_name cql_type )&lt;em&gt;; or DROP column_name ( column_name )&lt;/em&gt;; or WITH options&lt;/p&gt;
1911&lt;p&gt;The example associated to this statement is to ADD a new column to the table “species_table”, called “extinct” with type “boolean”.&lt;/p&gt;
1912&lt;pre&gt;&lt;code&gt;**ALTER** **TABLE** species_table **ADD** extinct **boolean**;
1913&lt;/code&gt;&lt;/pre&gt;
1914&lt;p&gt;Another example is to DROP the column called “sex” from the table “species_table”.&lt;/p&gt;
1915&lt;pre&gt;&lt;code&gt;**ALTER** **TABLE** species_table **DROP** sex;
1916&lt;/code&gt;&lt;/pre&gt;
1917&lt;p&gt;Finally, alter the comment with the WITH clause and set the comment to “All species records”. &lt;/p&gt;
1918&lt;pre&gt;&lt;code&gt;**ALTER** **TABLE** species_table **WITH** **comment**='All species records';
1919&lt;/code&gt;&lt;/pre&gt;
1920&lt;p&gt;These changes can be checked with the **DESCRIBE **statement:&lt;/p&gt;
1921&lt;pre&gt;&lt;code&gt;**DESCRIBE** **TABLE** species_table;
1922&lt;/code&gt;&lt;/pre&gt;
1923&lt;p&gt;&lt;img src=&quot;xebKPqkWkn97YVHpRVXZYWvRUfeRUyCH-vPDs67aFaEeU53YTRbDOFscOlAr.png&quot; alt=&quot;&quot; /&gt;
1924&lt;em&gt;DESCRIBE table&lt;/em&gt;&lt;/p&gt;
1925&lt;h3 id=&quot;dml_data_manipulation_language__2&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#dml_data_manipulation_language__2&quot;&gt;¶&lt;/a&gt;DML (Data Manipulation Language)&lt;/h3&gt;
1926&lt;h4 id=&quot;update_data&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#update_data&quot;&gt;¶&lt;/a&gt;Update data&lt;/h4&gt;
1927&lt;p&gt;Updating a row is done using an **UPDATE **statement:&lt;/p&gt;
1928&lt;pre&gt;&lt;code&gt;**UPDATE** table_name
1929[ **USING** update_parameter ( **AND** update_parameter )* ]
1930**SET** assignment ( ',' assignment )*
1931**WHERE** where_clause
1932[ **IF** ( **EXISTS** | condition ( **AND** condition )*) ];
1933&lt;/code&gt;&lt;/pre&gt;
1934&lt;p&gt;Where the update_parameter is: ( TIMESTAMP | TTL) (integer | bind_marker)&lt;/p&gt;
1935&lt;p&gt;It is important to mention that the **WHERE **clause is used to select the row to update and **must &lt;strong&gt;include ** all columns&lt;/strong&gt; composing the &lt;strong&gt;PRIMARY KEY.&lt;/strong&gt;&lt;/p&gt;
1936&lt;p&gt;We are going to test this statement updating the column “extinct” to true to the column with name ‘White monkey’.&lt;/p&gt;
1937&lt;pre&gt;&lt;code&gt;**UPDATE** species_table **SET** extinct = **true** **WHERE** species='White monkey';
1938&lt;/code&gt;&lt;/pre&gt;
1939&lt;p&gt;&lt;img src=&quot;IcaCe6VEC5c0ZQIygz-CiclzFyt491u7xPMg2muJLR8grmqaiUzkoQsVCoHf.png&quot; alt=&quot;&quot; /&gt;
1940&lt;em&gt;SELECT statement&lt;/em&gt;&lt;/p&gt;
1941&lt;h2 id=&quot;delete&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#delete&quot;&gt;¶&lt;/a&gt;Delete&lt;/h2&gt;
1942&lt;h3 id=&quot;ddl_data_definition_language__3&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#ddl_data_definition_language__3&quot;&gt;¶&lt;/a&gt;DDL (Data Definition Language)&lt;/h3&gt;
1943&lt;h4 id=&quot;drop_keyspace&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#drop_keyspace&quot;&gt;¶&lt;/a&gt;Drop keyspace&lt;/h4&gt;
1944&lt;p&gt;Dropping a keyspace can be done using the **DROP KEYSPACE **statement:&lt;/p&gt;
1945&lt;pre&gt;&lt;code&gt;**DROP** KEYSPACE [ **IF** **EXISTS** ] keyspace_name;
1946&lt;/code&gt;&lt;/pre&gt;
1947&lt;p&gt;For example, drop the keyspace called “test_keyspace_2” if it exists:&lt;/p&gt;
1948&lt;pre&gt;&lt;code&gt;**DROP** KEYSPACE **IF** **EXISTS** test_keyspace_2;
1949&lt;/code&gt;&lt;/pre&gt;
1950&lt;p&gt;As this keyspace does not exists, this sentence will do nothing.&lt;/p&gt;
1951&lt;h4 id=&quot;drop_table&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#drop_table&quot;&gt;¶&lt;/a&gt;Drop table&lt;/h4&gt;
1952&lt;p&gt;Dropping a table uses the **DROP TABLE **statement:&lt;/p&gt;
1953&lt;pre&gt;&lt;code&gt;**DROP** **TABLE** [ **IF** **EXISTS** ] table_name;
1954&lt;/code&gt;&lt;/pre&gt;
1955&lt;p&gt;For example, drop the table called “species_2” if it exists: &lt;/p&gt;
1956&lt;pre&gt;&lt;code&gt;**DROP** **TABLE** **IF** **EXISTS** species_2;
1957&lt;/code&gt;&lt;/pre&gt;
1958&lt;p&gt;As this table does not exists, this sentence will do nothing.&lt;/p&gt;
1959&lt;h4 id=&quot;truncate_table_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#truncate_table_&quot;&gt;¶&lt;/a&gt;Truncate (table)&lt;/h4&gt;
1960&lt;p&gt;A table can be truncated using the **TRUNCATE **statement:&lt;/p&gt;
1961&lt;pre&gt;&lt;code&gt;**TRUNCATE** [ **TABLE** ] table_name;
1962&lt;/code&gt;&lt;/pre&gt;
1963&lt;p&gt;Do not execute this command now, because if you do it, you will need to insert the previous data again.&lt;/p&gt;
1964&lt;p&gt;Note: as tables are the only object that can be truncated the keyword TABLE can be omitted.&lt;/p&gt;
1965&lt;p&gt;&lt;img src=&quot;FOkhfpxlWFQCzcdfeWxLTy7wx5inDv0xwVeVhE79Pqtk3yYzWsZJnz_SBhUi.png&quot; alt=&quot;&quot; /&gt;
1966&lt;em&gt;TRUNCATE statement&lt;/em&gt;&lt;/p&gt;
1967&lt;h3 id=&quot;dml_data_manipulation_language__3&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#dml_data_manipulation_language__3&quot;&gt;¶&lt;/a&gt;DML (Data Manipulation Language)&lt;/h3&gt;
1968&lt;h4 id=&quot;delete_data&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#delete_data&quot;&gt;¶&lt;/a&gt;Delete data&lt;/h4&gt;
1969&lt;p&gt;Deleting rows or parts of rows uses the **DELETE **statement:&lt;/p&gt;
1970&lt;pre&gt;&lt;code&gt;**DELETE** [ simple_selection ( ',' simple_selection ) ]
1971                      **FROM** table_name
1972                      [ **USING** update_parameter ( **AND** update_parameter )* ]
1973                      **WHERE** where_clause
1974                      [ **IF** ( **EXISTS** | condition ( **AND** condition )*) ]
1975&lt;/code&gt;&lt;/pre&gt;
1976&lt;p&gt;Now we are going to delete the value of the column “average_size” from “Cloned Sheep”. &lt;/p&gt;
1977&lt;pre&gt;&lt;code&gt;**DELETE** average_size **FROM** species_table **WHERE** species = 'Cloned Sheep';
1978&lt;/code&gt;&lt;/pre&gt;
1979&lt;p&gt;&lt;img src=&quot;CyuQokVL5J9TAelq-WEWhNl6kFtbIYs0R1AeU5NX4EkG-YQI81mNHdnf2yWN.png&quot; alt=&quot;&quot; /&gt;
1980&lt;em&gt;DELETE value statement&lt;/em&gt;&lt;/p&gt;
1981&lt;p&gt;And we are going to delete the same row as mentioned before. &lt;/p&gt;
1982&lt;pre&gt;&lt;code&gt;**DELETE** **FROM** species_table **WHERE** species = 'Cloned Sheep';
1983&lt;/code&gt;&lt;/pre&gt;
1984&lt;p&gt;&lt;img src=&quot;jvQ5cXJ5GTVQ6giVhBEpPJmrJw-zwKKyB9nsTm5PRcGSTzkmh-WO4kTeuLpB.png&quot; alt=&quot;&quot; /&gt;
1985&lt;em&gt;DELETE row statement&lt;/em&gt;&lt;/p&gt;
1986&lt;h2 id=&quot;batch&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#batch&quot;&gt;¶&lt;/a&gt;Batch&lt;/h2&gt;
1987&lt;p&gt;Multiple &lt;strong&gt;INSERT&lt;/strong&gt;, **UPDATE **and **DELETE **can be executed in a &lt;strong&gt;single statement&lt;/strong&gt; by grouping them through a **BATCH **statement.&lt;/p&gt;
1988&lt;pre&gt;&lt;code&gt;**BEGIN** [ UNLOGGED | COUNTER ] BATCH
1989                            [ **USING** update_parameter ( **AND** update_parameter )* ]
1990                            modification_statement ( ';' modification_statement )*
1991                            APPLY BATCH;
1992&lt;/code&gt;&lt;/pre&gt;
1993&lt;p&gt;Where modification_statement can be a insert_statement or an update_statement or a delete_statement.&lt;/p&gt;
1994&lt;ul&gt;
1995&lt;li&gt;**UNLOGGED **means that either all operations in a batch eventually complete or none will.&lt;/li&gt;
1996&lt;li&gt;&lt;strong&gt;COUNTER&lt;/strong&gt; means that the updates are not idempotent, so each time we execute the updates in a batch, we will have different results.
1997For example:&lt;/li&gt;
1998&lt;/ul&gt;
1999&lt;pre&gt;&lt;code&gt;**BEGIN** BATCH
2000   **INSERT** **INTO** species_table (species, common_name, population, average_size, extinct)
2001                **VALUES** ('Blue Shark', 'Tiburillo', 30, 10, **false**);
2002   **INSERT** **INTO** species_table (species, common_name, population, average_size, extinct)
2003                **VALUES** ('Cloned sheep', 'Dolly the Sheep', 1, 1, **true**);
2004   **UPDATE** species_table **SET** population = 2 **WHERE** species='Cloned sheep';
2005   **DELETE** **FROM** species_table **WHERE** species =  'White monkey';
2006APPLY BATCH;
2007&lt;/code&gt;&lt;/pre&gt;
2008&lt;p&gt;&lt;img src=&quot;EL9Dac26o0FqkVoeAKmopEKQe0wWq-xYI14b9RzGxtUkFJA3i2eTiR6qkuuJ.png&quot; alt=&quot;&quot; /&gt;
2009&lt;em&gt;BATCH statement&lt;/em&gt;&lt;/p&gt;
2010&lt;h2 id=&quot;index&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#index&quot;&gt;¶&lt;/a&gt;Index&lt;/h2&gt;
2011&lt;p&gt;CQL support creating secondary indexes on tables, allowing queries on the table to use those indexes. &lt;/p&gt;
2012&lt;p&gt;**Creating **a secondary index on a table uses the **CREATE INDEX **statement:&lt;/p&gt;
2013&lt;pre&gt;&lt;code&gt;**CREATE** [ CUSTOM ] **INDEX** [ **IF** **NOT** **EXISTS** ] [ index_name ]
2014                                **ON** table_name '(' index_identifier ')'
2015                                [ **USING** string [ **WITH** OPTIONS = map_literal ] ];
2016&lt;/code&gt;&lt;/pre&gt;
2017&lt;p&gt;For example we are going to create a index called “population_idx” that is related to the column “population” in the table “species_table”.&lt;/p&gt;
2018&lt;pre&gt;&lt;code&gt;**CREATE** **INDEX** population_idx **ON** species_table (population);
2019&lt;/code&gt;&lt;/pre&gt;
2020&lt;p&gt;**Dropping **a secondary index uses the &lt;strong&gt;DROP INDEX&lt;/strong&gt; statement: &lt;/p&gt;
2021&lt;pre&gt;&lt;code&gt;**DROP** **INDEX** [ **IF** **EXISTS** ] index_name;
2022&lt;/code&gt;&lt;/pre&gt;
2023&lt;p&gt;Now, we are going to drop the previous index: &lt;/p&gt;
2024&lt;pre&gt;&lt;code&gt;**DROP** **INDEX** **IF** **EXISTS** population_idx;
2025&lt;/code&gt;&lt;/pre&gt;
2026&lt;h2 id=&quot;references&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#references&quot;&gt;¶&lt;/a&gt;References&lt;/h2&gt;
2027&lt;ul&gt;
2028&lt;li&gt;&lt;a href=&quot;https://cassandra.apache.org/doc/latest/cql/ddl.html&quot;&gt;Cassandra CQL&lt;/a&gt;&lt;/li&gt;
2029&lt;li&gt;&lt;a href=&quot;https://techdifferences.com/difference-between-ddl-and-dml-in-dbms.html&quot;&gt;Differences between DML and DDL&lt;/a&gt;&lt;/li&gt;
2030&lt;li&gt;&lt;a href=&quot;https://docs.datastax.com/en/dse/5.1/cql/cql/cql_reference/cqlReferenceTOC.html&quot;&gt;Datastax CQL&lt;/a&gt;&lt;/li&gt;
2031&lt;li&gt;&lt;a href=&quot;https://cassandra.apache.org/doc/latest/cql/types.html#grammar-token-cql-type&quot;&gt;Cassandra CQL Types&lt;/a&gt;&lt;/li&gt;
2032&lt;li&gt;&lt;a href=&quot;https://cassandra.apache.org/doc/latest/cql/indexes.html&quot;&gt;Cassandra Index&lt;/a&gt;&lt;/li&gt;
2033&lt;/ul&gt;
2034&lt;/main&gt;
2035&lt;/body&gt;
2036&lt;/html&gt;
2037 </content></entry><entry><title>Upgrading our Baby Crawler</title><id>dist/upgrading-our-baby-crawler/index.html</id><updated>2020-03-17T23:00:00+00:00</updated><published>2020-03-10T23:00:00+00:00</published><summary>In our </summary><content type="html" src="dist/upgrading-our-baby-crawler/index.html">&lt;!DOCTYPE html&gt;
2038&lt;html&gt;
2039&lt;head&gt;
2040&lt;meta charset=&quot;utf-8&quot; /&gt;
2041&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
2042&lt;title&gt;Upgrading our Baby Crawler&lt;/title&gt;
2043&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
2044&lt;/head&gt;
2045&lt;body&gt;
2046&lt;main&gt;
2047&lt;p&gt;In our &lt;a href=&quot;/blog/ribw/build-your-own-pc/&quot;&gt;last post on this series&lt;/a&gt;, we presented the code for our Personal Crawler. However, we didn’t quite explain what a crawler even is! We will use this moment to go a bit more in-depth, and make some upgrades to it.&lt;/p&gt;
2048&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-11&lt;br&gt;
2049Modified 2020-03-18&lt;/div&gt;
2050&lt;h2 class=&quot;title&quot; id=&quot;what_is_a_crawler_&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#what_is_a_crawler_&quot;&gt;¶&lt;/a&gt;What is a Crawler?&lt;/h2&gt;
2051&lt;p&gt;A crawler is a program whose job is to analyze documents and extract data from them. For example, search engines like &lt;a href=&quot;http://duckduckgo.com/&quot;&gt;DuckDuckGo&lt;/a&gt;, &lt;a href=&quot;https://bing.com/&quot;&gt;Bing&lt;/a&gt; or &lt;a href=&quot;http://google.com/&quot;&gt;Google&lt;/a&gt; all have crawlers to analyze websites and build a database around them. They are some kind of «trackers», because they keep track of everything they find.&lt;/p&gt;
2052&lt;p&gt;Their basic behaviour can be described as follows: given a starting list of URLs, follow them all and identify hyperlinks inside the documents. Add these to the list of links to follow, and repeat &lt;em&gt;ad infinitum&lt;/em&gt;.&lt;/p&gt;
2053&lt;ul&gt;
2054&lt;li&gt;This lets us create an index to quickly search across them all.&lt;/li&gt;
2055&lt;li&gt;We can also identify broken links.&lt;/li&gt;
2056&lt;li&gt;We can gather any other type of information that we found.
2057Our crawler will work offline, within our own computer, scanning the text documents it finds on the root we tell it to scan.&lt;/li&gt;
2058&lt;/ul&gt;
2059&lt;h2 id=&quot;design_decissions&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#design_decissions&quot;&gt;¶&lt;/a&gt;Design Decissions&lt;/h2&gt;
2060&lt;ul&gt;
2061&lt;li&gt;We will use Java. Its runtime is quite ubiquitous, so it should be able to run in virtually anywhere. The language is typed, which helps catch errors early on.&lt;/li&gt;
2062&lt;li&gt;Our solution is iterative. While recursion can be seen as more elegants by some, iterative solutions are often more performant with less need for optimization.&lt;/li&gt;
2063&lt;/ul&gt;
2064&lt;h2 id=&quot;requirements&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#requirements&quot;&gt;¶&lt;/a&gt;Requirements&lt;/h2&gt;
2065&lt;p&gt;If you don’t have Java installed yet, you can &lt;a href=&quot;https://java.com/en/download/&quot;&gt;Download Free Java Software&lt;/a&gt; from Oracle’s site. To compile the code, the &lt;a href=&quot;https://www.oracle.com/java/technologies/javase-jdk8-downloads.html&quot;&gt;Java Development Kit&lt;/a&gt; is also necessary.&lt;/p&gt;
2066&lt;p&gt;We don’t depend on any other external libraries, for easier deployment and compilation.&lt;/p&gt;
2067&lt;h2 id=&quot;implementation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#implementation&quot;&gt;¶&lt;/a&gt;Implementation&lt;/h2&gt;
2068&lt;p&gt;Because the code was getting pretty large, it has been split into several files, and we have also upgraded it to use a Graphical User Interface instead! We decided to use Swing, based on the Java tutorial &lt;a href=&quot;https://docs.oracle.com/javase/tutorial/uiswing/&quot;&gt;Creating a GUI With JFC/Swing&lt;/a&gt;.&lt;/p&gt;
2069&lt;h3 id=&quot;app&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#app&quot;&gt;¶&lt;/a&gt;App&lt;/h3&gt;
2070&lt;p&gt;This file is the entry point of our application. Its job is to initialize the components, lay them out in the main panel, and connect the event handlers.&lt;/p&gt;
2071&lt;p&gt;Most widgets are pretty standard, and are defined as class variables. However, some variables are notable. The &lt;code&gt;[DefaultTableModel](https://docs.oracle.com/javase/8/docs/api/javax/swing/table/DefaultTableModel.html)&lt;/code&gt; is used because it allows to &lt;a href=&quot;https://stackoverflow.com/a/22550106&quot;&gt;dynamically add rows&lt;/a&gt;, and we also have a &lt;code&gt;[SwingWorker](https://docs.oracle.com/javase/8/docs/api/javax/swing/SwingWorker.html)&lt;/code&gt; subclass responsible for performing the word analysis (which is quite CPU intensive and should not be ran in the UI thread!).&lt;/p&gt;
2072&lt;p&gt;There’s a few utility methods to ease some common operations, such as &lt;code&gt;updateStatus&lt;/code&gt; which changes the status label in the main window, informing the user of the latest changes.&lt;/p&gt;
2073&lt;h3 id=&quot;thesaurus&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#thesaurus&quot;&gt;¶&lt;/a&gt;Thesaurus&lt;/h3&gt;
2074&lt;p&gt;A thesaurus is a collection of words or terms used to represent concepts. In literature this is commonly known as a dictionary.&lt;/p&gt;
2075&lt;p&gt;On the subject of this project, we are using a thesaurus based on how relevant is a word for the meaning of a sentence, filtering out those that barely give us any information.&lt;/p&gt;
2076&lt;p&gt;This file contains a simple thesaurus implementation, which can trivially be used as a normal or inverted thesaurus. However, we only treat it as inverted, and its job is loading itself and determining if words are valid or should otherwise be ignored.&lt;/p&gt;
2077&lt;h3 id=&quot;utils&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#utils&quot;&gt;¶&lt;/a&gt;Utils&lt;/h3&gt;
2078&lt;p&gt;Several utility functions used across the codebase.&lt;/p&gt;
2079&lt;h3 id=&quot;wordmap&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#wordmap&quot;&gt;¶&lt;/a&gt;WordMap&lt;/h3&gt;
2080&lt;p&gt;This file is the important one, and its implementation hasn’t changed much since our last post. Instances of a word map contain… wait for it… a map of words! It stores the mapping &lt;code&gt;word → count&lt;/code&gt; in memory, and offers methods to query the count of a word or iterate over the word count entries.&lt;/p&gt;
2081&lt;p&gt;It can be loaded from cache or told to analyze a root path. Once an instance is created, additional files could be analyzed one by one if desired.&lt;/p&gt;
2082&lt;h2 id=&quot;download&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#download&quot;&gt;¶&lt;/a&gt;Download&lt;/h2&gt;
2083&lt;p&gt;The code was getting a bit too large to embed it within the blog post itself, so instead you can download it as a&lt;code&gt;.zip&lt;/code&gt; file.&lt;/p&gt;
2084&lt;p&gt;&lt;em&gt;download removed&lt;/em&gt;&lt;/p&gt;
2085&lt;/main&gt;
2086&lt;/body&gt;
2087&lt;/html&gt;
2088 </content></entry><entry><title>Cassandra: an Introduction</title><id>dist/cassandra-an-introduction/index.html</id><updated>2020-03-17T23:00:00+00:00</updated><published>2020-03-04T23:00:00+00:00</published><summary>This is the first post in the Cassandra series, where we will introduce the Cassandra database system and take a look at its features and installation methods.</summary><content type="html" src="dist/cassandra-an-introduction/index.html">&lt;!DOCTYPE html&gt;
2089&lt;html&gt;
2090&lt;head&gt;
2091&lt;meta charset=&quot;utf-8&quot; /&gt;
2092&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
2093&lt;title&gt;Cassandra: an Introduction&lt;/title&gt;
2094&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
2095&lt;/head&gt;
2096&lt;body&gt;
2097&lt;main&gt;
2098&lt;p&gt;This is the first post in the Cassandra series, where we will introduce the Cassandra database system and take a look at its features and installation methods.&lt;/p&gt;
2099&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-05&lt;br&gt;
2100Modified 2020-03-18&lt;/div&gt;
2101&lt;p&gt;Other posts in this series:&lt;/p&gt;
2102&lt;ul&gt;
2103&lt;li&gt;&lt;a href=&quot;/blog/ribw/cassandra-an-introduction/&quot;&gt;Cassandra: an Introduction&lt;/a&gt; (this post)&lt;/li&gt;
2104&lt;/ul&gt;
2105&lt;p&gt;This post is co-authored wih Classmate.&lt;/p&gt;
2106&lt;hr /&gt;
2107&lt;div class=&quot;image-container&quot;&gt;
2108&lt;img src=&quot;cassandra-database-e1584191543401.jpg&quot; alt=&quot;NoSQL database – Apache Cassandra – First delivery&quot; /&gt;
2109&lt;div class=&quot;image-caption&quot;&gt;&lt;/div&gt;
2110&lt;/div&gt;
2111&lt;p&gt;
2112&lt;h2 class=&quot;title&quot; id=&quot;purpose_of_technology&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#purpose_of_technology&quot;&gt;¶&lt;/a&gt;Purpose of technology&lt;/h2&gt;
2113&lt;p&gt;Apache Cassandra is a &lt;strong&gt;NoSQL&lt;/strong&gt;, &lt;strong&gt;open-source&lt;/strong&gt;, &lt;strong&gt;distributed “key-value” database&lt;/strong&gt;. It allows &lt;strong&gt;large volumes of distributed data&lt;/strong&gt;. The main **goal **is provide &lt;strong&gt;linear scalability and availabilitywithout compromising performance&lt;/strong&gt;. Besides, Cassandra &lt;strong&gt;supports replication&lt;/strong&gt; across multiple datacenters, providing low latency. &lt;/p&gt;
2114&lt;h2 id=&quot;how_it_works&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#how_it_works&quot;&gt;¶&lt;/a&gt;How it works&lt;/h2&gt;
2115&lt;p&gt;Cassandra’s distributed **architecture **is based on a series of &lt;strong&gt;equal nodes&lt;/strong&gt; that communicate with a &lt;strong&gt;P2P protocol&lt;/strong&gt; so that &lt;strong&gt;redundancy is maximum&lt;/strong&gt;. It offers robust support for multiple datacenters, with &lt;strong&gt;asynchronous replication&lt;/strong&gt; without the need for a master server. &lt;/p&gt;
2116&lt;p&gt;Besides, Cassandra’s &lt;strong&gt;data model consists of partitioning the rows&lt;/strong&gt;, which are rearranged into &lt;strong&gt;different tables&lt;/strong&gt;. The primary keys of each table have a first component that is the &lt;strong&gt;partition key&lt;/strong&gt;. Within a partition, the rows are grouped by the remaining columns of the key. The other columns can be indexed separately from the primary key.&lt;/p&gt;
2117&lt;p&gt;These tables can be &lt;strong&gt;created, deleted, updated and queried****at runtime without blocking&lt;/strong&gt; each other. However it does &lt;strong&gt;not support joins or subqueries&lt;/strong&gt;, but instead &lt;strong&gt;emphasizes denormalization&lt;/strong&gt; through features like collections.&lt;/p&gt;
2118&lt;p&gt;Nowadays, Cassandra uses its own query language called &lt;strong&gt;CQL&lt;/strong&gt; (&lt;strong&gt;Cassandra Query Language&lt;/strong&gt;), with a &lt;strong&gt;similar syntax to SQL&lt;/strong&gt;. It also allows access from &lt;strong&gt;JDBC&lt;/strong&gt;.&lt;/p&gt;
2119&lt;p&gt;&lt;img src=&quot;s0GHpggGZXOFcdhypRWV4trU-PkSI6lukEv54pLZnoirh0GlDVAc4LamB1Dy.png&quot; alt=&quot;&quot; /&gt;
2120_ Cassandra architecture _&lt;/p&gt;
2121&lt;h2 id=&quot;features&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#features&quot;&gt;¶&lt;/a&gt;Features&lt;/h2&gt;
2122&lt;ul&gt;
2123&lt;li&gt;&lt;strong&gt;Decentralized&lt;/strong&gt;: there are &lt;strong&gt;no single points of failure&lt;/strong&gt;, every **node **in the cluster has the &lt;strong&gt;same role&lt;/strong&gt; and there is &lt;strong&gt;no master node&lt;/strong&gt;, so each node &lt;strong&gt;can service any request&lt;/strong&gt;, besides the data is distributed across the cluster.&lt;/li&gt;
2124&lt;li&gt;Supports **replication **and multiple replication of &lt;strong&gt;data center&lt;/strong&gt;: the replication strategies are &lt;strong&gt;configurable&lt;/strong&gt;. &lt;/li&gt;
2125&lt;li&gt;**Scalability: **reading and writing performance increases linearly as new nodes are added, also &lt;strong&gt;new nodes&lt;/strong&gt; can be &lt;strong&gt;added without interrupting&lt;/strong&gt; application &lt;strong&gt;execution&lt;/strong&gt;.&lt;/li&gt;
2126&lt;li&gt;&lt;strong&gt;Fault tolerance: data replication&lt;/strong&gt; is done **automatically **in several nodes in order to recover from failures. It is possible to &lt;strong&gt;replace failure nodes****without &lt;strong&gt;making&lt;/strong&gt; inactivity time or interruptions&lt;/strong&gt; to the application.&lt;/li&gt;
2127&lt;li&gt;**Consistency: **a choice of consistency level is provided for &lt;strong&gt;reading and writing&lt;/strong&gt;.&lt;/li&gt;
2128&lt;li&gt;&lt;strong&gt;MapReduce support&lt;/strong&gt;: it is **integrated **with &lt;strong&gt;Apache Hadoop&lt;/strong&gt; to support MapReduce.&lt;/li&gt;
2129&lt;li&gt;&lt;strong&gt;Query language&lt;/strong&gt;: it has its own query language called **CQL (Cassandra Query Language) **&lt;/li&gt;
2130&lt;/ul&gt;
2131&lt;h2 id=&quot;corner_in_cap_theorem&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#corner_in_cap_theorem&quot;&gt;¶&lt;/a&gt;Corner in CAP theorem&lt;/h2&gt;
2132&lt;p&gt;&lt;strong&gt;Apache Cassandra&lt;/strong&gt; is usually described as an “&lt;strong&gt;AP&lt;/strong&gt;” system because it guarantees &lt;strong&gt;availability&lt;/strong&gt; and &lt;strong&gt;partition/fault tolerance&lt;/strong&gt;. So it errs on the side of ensuring data availability even if this means &lt;strong&gt;sacrificing consistency&lt;/strong&gt;. But, despite this fact, Apache Cassandra &lt;strong&gt;seeks to satisfy all three requirements&lt;/strong&gt; (Consistency, Availability and Fault tolerance) simultaneously and can be &lt;strong&gt;configured to behave&lt;/strong&gt; like a “&lt;strong&gt;CP&lt;/strong&gt;” database, guaranteeing &lt;strong&gt;consistency and partition/fault tolerance&lt;/strong&gt;. &lt;/p&gt;
2133&lt;p&gt;&lt;img src=&quot;rf3n9LTOKCQVbx4qrn7NPSVcRcwE1LxR_khi-9Qc51Hcbg6BHHPu-0GZjUwD.png&quot; alt=&quot;&quot; /&gt;
2134&lt;em&gt;Cassandra in CAP Theorem&lt;/em&gt;&lt;/p&gt;
2135&lt;h2 id=&quot;download&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#download&quot;&gt;¶&lt;/a&gt;Download&lt;/h2&gt;
2136&lt;p&gt;In order to download the file, with extension .tar.gz. you must visit the &lt;a href=&quot;https://cassandra.apache.org/download/&quot;&gt;download site&lt;/a&gt; and click on the file “&lt;a href=&quot;https://ftp.cixug.es/apache/cassandra/3.11.6/apache-cassandra-3.11.6-bin.tar.gz&quot;&gt;https://ftp.cixug.es/apache/cassandra/3.11.6/apache-cassandra-3.11.6-bin.tar.gz&lt;/a&gt;”. It is important to mention that the previous link is related to the 3.11.6 version.&lt;/p&gt;
2137&lt;h2 id=&quot;installation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#installation&quot;&gt;¶&lt;/a&gt;Installation&lt;/h2&gt;
2138&lt;p&gt;This database can only be installed on Linux distributions and Mac OS X systems, so, it is not possible to install it on Microsoft Windows.&lt;/p&gt;
2139&lt;p&gt;The first main requirement is having installed Java 8 in &lt;strong&gt;Ubuntu&lt;/strong&gt;, the OS that we will use. Therefore, the Java 8 installation is explained below. First open a terminal and execute the next command:&lt;/p&gt;
2140&lt;pre&gt;&lt;code&gt;sudo apt update
2141sudo apt install openjdk-8-jdk openjdk-8-jre
2142&lt;/code&gt;&lt;/pre&gt;
2143&lt;p&gt;In order to establish Java as a environment variable it is needed to open the file “/.bashrc”: &lt;/p&gt;
2144&lt;pre&gt;&lt;code&gt;nano ~/.bashrc
2145&lt;/code&gt;&lt;/pre&gt;
2146&lt;p&gt;And add at the end of it the path where Java is installed, as follows: &lt;/p&gt;
2147&lt;pre&gt;&lt;code&gt;export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
2148export PATH=$PATH:$JAVA_HOME/bin
2149&lt;/code&gt;&lt;/pre&gt;
2150&lt;p&gt;At this point, save the file and execute the next command, note that it does the same effect re-opening the terminal: &lt;/p&gt;
2151&lt;pre&gt;&lt;code&gt;source ~/.bashrc
2152&lt;/code&gt;&lt;/pre&gt;
2153&lt;p&gt;In order to check if the Java environment variable is set correctly, run the next command: &lt;/p&gt;
2154&lt;pre&gt;&lt;code&gt;echo $JAVA_HOME
2155&lt;/code&gt;&lt;/pre&gt;
2156&lt;p&gt;&lt;img src=&quot;JUUmX5MIHynJR_K9EdCgKeJcpINeCGRRt2QRu4JLPtRhCVidOhcbWwVTQjyu.png&quot; alt=&quot;&quot; /&gt;
2157&lt;em&gt;$JAVAHOME variable&lt;/em&gt;&lt;/p&gt;
2158&lt;p&gt;Afterwards, it is possible to check the installed Java version with the command: &lt;/p&gt;
2159&lt;pre&gt;&lt;code&gt;java -version
2160&lt;/code&gt;&lt;/pre&gt;
2161&lt;p&gt;&lt;img src=&quot;z9v1-0hpZwjI4U5UZej9cRGN5-Y4AZl0WUPWyQ_-JlzTAIvZtTFPnKY2xMQ_.png&quot; alt=&quot;&quot; /&gt;
2162&lt;em&gt;Java version&lt;/em&gt;&lt;/p&gt;
2163&lt;p&gt;The next requirement is having installed the latest version of Python 2.7. This can be checked with the command: &lt;/p&gt;
2164&lt;pre&gt;&lt;code&gt;python --version
2165&lt;/code&gt;&lt;/pre&gt;
2166&lt;p&gt;If it is not installed, to install it, it is as simple as run the next command in the terminal: &lt;/p&gt;
2167&lt;pre&gt;&lt;code&gt;sudo apt install python
2168&lt;/code&gt;&lt;/pre&gt;
2169&lt;p&gt;Note: it is better to use “python2” instead of “python” because in that way, you force to user Python 2.7. Modern distributions will use Python 3 for the «python» command.&lt;/p&gt;
2170&lt;p&gt;Therefore, it is possible to check the installed Python version with the command:&lt;/p&gt;
2171&lt;pre&gt;&lt;code&gt;python --version
2172&lt;/code&gt;&lt;/pre&gt;
2173&lt;p&gt;&lt;img src=&quot;Ger5Vw_e1HIK84QgRub-BwGmzIGKasgiYb4jHdfRNRrvG4d6Msp_3Vk62-9i.png&quot; alt=&quot;&quot; /&gt;
2174&lt;em&gt;Python version&lt;/em&gt;&lt;/p&gt;
2175&lt;p&gt;Once both requirements are ready, next step is to unzip the file previously downloaded, right click on the file and select “Extract here” or with the next command, on the directory where is the downloaded file. &lt;/p&gt;
2176&lt;pre&gt;&lt;code&gt;tar -zxvf apache-cassandra-x.x.x-bin.tar.gz
2177&lt;/code&gt;&lt;/pre&gt;
2178&lt;p&gt;In order to check if the installation is completed, you can execute the next command, in the root folder of the project. This will start Cassandra in a single node. &lt;/p&gt;
2179&lt;pre&gt;&lt;code&gt;/bin/cassandra
2180&lt;/code&gt;&lt;/pre&gt;
2181&lt;p&gt;It is possible to make a get some data from Cassandra with CQL (Cassandra Query Language). To check this execute the next command in another terminal. &lt;/p&gt;
2182&lt;pre&gt;&lt;code&gt;/bin/cqlsh localhost
2183&lt;/code&gt;&lt;/pre&gt;
2184&lt;p&gt;Once CQL is open, type the next sentence and check the result: &lt;/p&gt;
2185&lt;pre&gt;&lt;code&gt;SELECT cluster_name, listen_address from system.local;
2186&lt;/code&gt;&lt;/pre&gt;
2187&lt;p&gt;The output should be:&lt;/p&gt;
2188&lt;p&gt;&lt;img src=&quot;miUO60A-RtyEAOOVFJqlkPRC18H4RKUhot6RWzhO9FmtzgTPOYHFtwxqgZEf.png&quot; alt=&quot;&quot; /&gt;
2189&lt;em&gt;Sentence output&lt;/em&gt;&lt;/p&gt;
2190&lt;p&gt;Finally, the installation guide provided by the website of the database is attached in this &lt;a href=&quot;https://cassandra.apache.org/doc/latest/getting_started/installing.html&quot;&gt;installation guide&lt;/a&gt;. &lt;/p&gt;
2191&lt;h2 id=&quot;references&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#references&quot;&gt;¶&lt;/a&gt;References&lt;/h2&gt;
2192&lt;ul&gt;
2193&lt;li&gt;&lt;a href=&quot;https://es.wikipedia.org/wiki/Apache_Cassandra&quot;&gt;Wikipedia&lt;/a&gt;&lt;/li&gt;
2194&lt;li&gt;&lt;a href=&quot;https://cassandra.apache.org/&quot;&gt;Apache Cassandra&lt;/a&gt;&lt;/li&gt;
2195&lt;li&gt;&lt;a href=&quot;https://www.datastax.com/blog/2019/05/how-apache-cassandratm-balances-consistency-availability-and-performance&quot;&gt;Datastax&lt;/a&gt;&lt;/li&gt;
2196&lt;li&gt;&lt;a href=&quot;https://blog.yugabyte.com/apache-cassandra-architecture-how-it-works-lightweight-transactions/&quot;&gt;yugabyte&lt;/a&gt;&lt;/li&gt;
2197&lt;/ul&gt;
2198&lt;/main&gt;
2199&lt;/body&gt;
2200&lt;/html&gt;
2201 </content></entry><entry><title>Privado: PC-Crawler evaluation</title><id>dist/pc-crawler-evaluation/index.html</id><updated>2020-03-17T23:00:00+00:00</updated><published>2020-03-03T23:00:00+00:00</published><summary>As the student </summary><content type="html" src="dist/pc-crawler-evaluation/index.html">&lt;!DOCTYPE html&gt;
2202&lt;html&gt;
2203&lt;head&gt;
2204&lt;meta charset=&quot;utf-8&quot; /&gt;
2205&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
2206&lt;title&gt;Privado: PC-Crawler evaluation&lt;/title&gt;
2207&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
2208&lt;/head&gt;
2209&lt;body&gt;
2210&lt;main&gt;
2211&lt;p&gt;As the student &lt;code&gt;a(i)&lt;/code&gt; where &lt;code&gt;i = 9&lt;/code&gt;, I have been assigned to evaluate students &lt;code&gt;a(i + 3)&lt;/code&gt; and &lt;code&gt;a(i + 4)&lt;/code&gt;, these being:&lt;/p&gt;
2212&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-03-04&lt;br&gt;
2213Modified 2020-03-18&lt;/div&gt;
2214&lt;ul&gt;
2215&lt;li&gt;a12: Classmate (username)&lt;/li&gt;
2216&lt;li&gt;a13: Classmate (username)&lt;/li&gt;
2217&lt;/ul&gt;
2218&lt;h2 class=&quot;title&quot; id=&quot;classmate_s_evaluation&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#classmate_s_evaluation&quot;&gt;¶&lt;/a&gt;Classmate’s evaluation&lt;/h2&gt;
2219&lt;p&gt;&lt;strong&gt;Grading: B.&lt;/strong&gt;&lt;/p&gt;
2220&lt;p&gt;I think they mix up a bit their considerations with program usage and how it works, not justifying why the considerations are the ones they chose, or what the alternatives would be.&lt;/p&gt;
2221&lt;p&gt;The implementation notes are quite well-written. Even someone without knowledge of Java’s syntax can read the notes and more or less make sense of what’s going on, with the relevant code excerpts on each section.&lt;/p&gt;
2222&lt;p&gt;Implementation-wise, some methods could definitely use some improvement:&lt;/p&gt;
2223&lt;ul&gt;
2224&lt;li&gt;&lt;code&gt;esExtensionTextual&lt;/code&gt; is overly complicated. It could use a &lt;code&gt;for&lt;/code&gt; loop and Java’s &lt;code&gt;String.endsWith&lt;/code&gt;.&lt;/li&gt;
2225&lt;li&gt;&lt;code&gt;calcularFrecuencia&lt;/code&gt; has quite some duplication (e.g. &lt;code&gt;this.getFicherosYDirectorios().remove(0)&lt;/code&gt;) and could definitely be cleaned up.&lt;/li&gt;
2226&lt;/ul&gt;
2227&lt;p&gt;However, all the desired functionality is implemented.&lt;/p&gt;
2228&lt;p&gt;Style-wise, some of the newlines and avoiding braces on &lt;code&gt;if&lt;/code&gt; and &lt;code&gt;while&lt;/code&gt; could be changed to improve the readability.&lt;/p&gt;
2229&lt;p&gt;The post is written in Spanish, but uses some words that don’t translate well («remover» could better be said as «eliminar» or «quitar»).&lt;/p&gt;
2230&lt;h2 id=&quot;classmate_s_evaluation_2&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#classmate_s_evaluation_2&quot;&gt;¶&lt;/a&gt;Classmate’s evaluation&lt;/h2&gt;
2231&lt;p&gt;&lt;strong&gt;Grading: B.&lt;/strong&gt;&lt;/p&gt;
2232&lt;p&gt;Their post starts with an explanation on what a crawler is, common uses for them, and what type of crawler they will be developing. This is a very good start. Regarding the post style, it seems they are not properly using some of WordPress features, such as lists, and instead rely on paragraphs with special characters prefixing each list item.&lt;/p&gt;
2233&lt;p&gt;The post also contains some details on how to install the requirements to run the program, which can be very useful for someone not used to working with Java.&lt;/p&gt;
2234&lt;p&gt;They do not explain their implementation and the filename of the download has a typo.&lt;/p&gt;
2235&lt;p&gt;Implementation-wise, the code seems to be well-organized, into several packages and files, although the naming is a bit inconsistent. They even designed a GUI, which is quite impressive.&lt;/p&gt;
2236&lt;p&gt;Some of the methods are documented, although the code inside them is not very commented, including missing rationale for the data structures chosen. There also seem to be several other unused main functions, which I’m unsure why they were kept.&lt;/p&gt;
2237&lt;p&gt;However, all the desired functionality is implemented.&lt;/p&gt;
2238&lt;p&gt;Similar to Classmate, the code style could be improved and settled on some standard, as well as making use of Java features such as &lt;code&gt;for&lt;/code&gt; loops over iterators instead of manual loops.&lt;/p&gt;
2239&lt;/main&gt;
2240&lt;/body&gt;
2241&lt;/html&gt;
2242 </content></entry><entry><title>Introduction to NoSQL</title><id>dist/introduction-to-nosql/index.html</id><updated>2020-03-17T23:00:00+00:00</updated><published>2020-02-24T23:00:00+00:00</published><summary>This post will primarly focus on the talk held in the </summary><content type="html" src="dist/introduction-to-nosql/index.html">&lt;!DOCTYPE html&gt;
2243&lt;html&gt;
2244&lt;head&gt;
2245&lt;meta charset=&quot;utf-8&quot; /&gt;
2246&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
2247&lt;title&gt;Introduction to NoSQL&lt;/title&gt;
2248&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
2249&lt;/head&gt;
2250&lt;body&gt;
2251&lt;main&gt;
2252&lt;p&gt;This post will primarly focus on the talk held in the &lt;a href=&quot;https://youtu.be/qI_g07C_Q5I&quot;&gt;GOTO 2012 conference: Introduction to NoSQL by Martin Fowler&lt;/a&gt;. It can be seen as an informal, summarized transcript of the talk&lt;/p&gt;
2253&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-02-25&lt;br&gt;
2254Modified 2020-03-18&lt;/div&gt;
2255&lt;hr /&gt;
2256&lt;p&gt;The relational database model is affected by the &lt;em&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Object-relational_impedance_mismatch&quot;&gt;impedance mismatch problem&lt;/a&gt;&lt;/em&gt;. This occurs because we have to match our high-level design with the separate columns and rows used by relational databases.&lt;/p&gt;
2257&lt;p&gt;Taking the in-memory objects and putting them into a relational database (which were dominant at the time) simply didn’t work out. Why? Relational databases were more than just databases, they served as a an integration mechanism across applications, up to the 2000s. For 20 years!&lt;/p&gt;
2258&lt;p&gt;With the rise of the Internet and the sheer amount of traffic, databases needed to scale. Unfortunately, relational databases only scale well vertically (by upgrading a &lt;em&gt;single&lt;/em&gt; node). This is &lt;em&gt;very&lt;/em&gt; expensive, and not something many could afford.&lt;/p&gt;
2259&lt;p&gt;The problem are those pesky &lt;code&gt;JOIN&lt;/code&gt;‘s, and its friends &lt;code&gt;GROUP BY&lt;/code&gt;. Because our program and reality model don’t match the tables used by SQL, we have to rely on them to query the data. It is because the model doesn’t map directly.&lt;/p&gt;
2260&lt;p&gt;Furthermore, graphs don’t map very well at all to relational models.&lt;/p&gt;
2261&lt;p&gt;We needed a way to scale horizontally (by increasing the &lt;em&gt;amount&lt;/em&gt; of nodes), something relational databases were not designed to do.&lt;/p&gt;
2262&lt;blockquote&gt;
2263&lt;p&gt;&lt;em&gt;We need to do something different, relational across nodes is an unnatural act&lt;/em&gt;&lt;/p&gt;
2264&lt;/blockquote&gt;
2265&lt;p&gt;This inspired the NoSQL movement.&lt;/p&gt;
2266&lt;blockquote&gt;
2267&lt;p&gt;&lt;em&gt;#nosql was only meant to be a hashtag to advertise it, but unfortunately it’s how it is called now&lt;/em&gt;&lt;/p&gt;
2268&lt;/blockquote&gt;
2269&lt;p&gt;It is not possible to define NoSQL, but we can identify some of its characteristics:&lt;/p&gt;
2270&lt;ul&gt;
2271&lt;li&gt;
2272&lt;p&gt;Non-relational&lt;/p&gt;
2273&lt;/li&gt;
2274&lt;li&gt;
2275&lt;p&gt;&lt;strong&gt;Cluster-friendly&lt;/strong&gt; (this was the original spark)&lt;/p&gt;
2276&lt;/li&gt;
2277&lt;li&gt;
2278&lt;p&gt;Open-source (until now, generally)&lt;/p&gt;
2279&lt;/li&gt;
2280&lt;li&gt;
2281&lt;p&gt;21st century web culture&lt;/p&gt;
2282&lt;/li&gt;
2283&lt;li&gt;
2284&lt;p&gt;Schema-less (easier integration or conjugation of several models, structure aggregation)
2285These databases use different data models to those used by the relational model. However, it is possible to identify 4 broad chunks (some may say 3, or even 2!):&lt;/p&gt;
2286&lt;/li&gt;
2287&lt;li&gt;
2288&lt;p&gt;&lt;strong&gt;Key-value store&lt;/strong&gt;. With a certain key, you obtain the value corresponding to it. It knows nothing else, nor does it care. We say the data is opaque.&lt;/p&gt;
2289&lt;/li&gt;
2290&lt;li&gt;
2291&lt;p&gt;&lt;strong&gt;Document-based&lt;/strong&gt;. It stores an entire mass of documents with complex structure, normally through the use of JSON (XML has been left behind). Then, you can ask for certain fields, structures, or portions. We say the data is transparent.&lt;/p&gt;
2292&lt;/li&gt;
2293&lt;li&gt;
2294&lt;p&gt;&lt;strong&gt;Column-family&lt;/strong&gt;. There is a «row key», and within it we store multiple «column families» (columns that fit together, our aggregate). We access by row-key and column-family name.
2295All of these kind of serve to store documents without any &lt;em&gt;explicit&lt;/em&gt; schema. Just shove in anything! This gives a lot of flexibility and ease of migration, except… that’s not really true. There’s an &lt;em&gt;implicit&lt;/em&gt; schema when querying.&lt;/p&gt;
2296&lt;/li&gt;
2297&lt;/ul&gt;
2298&lt;p&gt;For example, a query where we may do &lt;code&gt;anOrder['price'] * anOrder['quantity']&lt;/code&gt; is assuming that &lt;code&gt;anOrder&lt;/code&gt; has both a &lt;code&gt;price&lt;/code&gt; and a &lt;code&gt;quantity&lt;/code&gt;, and that both of these can be multiplied together. «Schema-less» is a fuzzy term.&lt;/p&gt;
2299&lt;p&gt;However, it is the lack of a &lt;em&gt;fixed&lt;/em&gt; schema that gives flexibility.&lt;/p&gt;
2300&lt;p&gt;One could argue that the line between key-value and document-based is very fuzzy, and they would be right! Key-value databases often let you include additional metadata that behaves like an index, and in document-based, documents often have an identifier anyway.&lt;/p&gt;
2301&lt;p&gt;The common notion between these three types is what matters. They save an entire structure as an &lt;em&gt;unit&lt;/em&gt;. We can refer to these as «Aggregate Oriented Databases». Aggregate, because we group things when designing or modeling our systems, as opposed to relational databases that scatter the information across many tables.&lt;/p&gt;
2302&lt;p&gt;There exists a notable outlier, though, and that’s:&lt;/p&gt;
2303&lt;ul&gt;
2304&lt;li&gt;&lt;strong&gt;Graph&lt;/strong&gt; databases. They use a node-and-arc graph structure. They are great for moving on relationships across things. Ironically, relational databases are not very good at jumping across relationships! It is possibly to perform very interesting queries in graph databases which would be really hard and costly on relational models. Unlike the aggregated databases, graphs break things into even smaller units.
2305NoSQL is not &lt;em&gt;the&lt;/em&gt; solution. It depends on how you’ll work with your data. Do you need an aggregate database? Will you have a lot of relationships? Or would the relational model be good fit for you?&lt;/li&gt;
2306&lt;/ul&gt;
2307&lt;p&gt;NoSQL, however, is a good fit for large-scale projects (data will &lt;em&gt;always&lt;/em&gt; grow) and faster development (the impedance mismatch is drastically reduced).&lt;/p&gt;
2308&lt;p&gt;Regardless of our choice, it is important to remember that NoSQL is a young technology, which is still evolving really fast (SQL has been stable for &lt;em&gt;decades&lt;/em&gt;). But the &lt;em&gt;polyglot persistence&lt;/em&gt; is what matters. One must know the alternatives, and be able to choose.&lt;/p&gt;
2309&lt;hr /&gt;
2310&lt;p&gt;Relational databases have the well-known ACID properties: Atomicity, Consistency, Isolation and Durability.&lt;/p&gt;
2311&lt;p&gt;NoSQL (except graph-based!) are about being BASE instead: Basically Available, Soft state, Eventual consistency.&lt;/p&gt;
2312&lt;p&gt;SQL needs transactions because we don’t want to perform a read while we’re only half-way done with a write! The readers and writers are the problem, and ensuring consistency results in a performance hit, even if the risk is low (two writers are extremely rare but it still must be handled).&lt;/p&gt;
2313&lt;p&gt;NoSQL on the other hand doesn’t need ACID because the aggregate &lt;em&gt;is&lt;/em&gt; the transaction boundary. Even before NoSQL itself existed! Any update is atomic by nature. When updating many documents it &lt;em&gt;is&lt;/em&gt; a problem, but this is very rare.&lt;/p&gt;
2314&lt;p&gt;We have to distinguish between logical and replication consistency. During an update and if a conflict occurs, it must be resolved to preserve the logical consistency. Replication consistency on the other hand is preserveed when distributing the data across many machines, for example during sharding or copies.&lt;/p&gt;
2315&lt;p&gt;Replication buys us more processing power and resillence (at the cost of more storage) in case some of the nodes die. But what happens if what dies is the communication across the nodes? We could drop the requests and preserve the consistency, or accept the risk to continue and instead preserve the availability.&lt;/p&gt;
2316&lt;p&gt;The choice on whether trading consistency for availability is acceptable or not depends on the domain rules. It is the domain’s choice, the business people will choose. If you’re Amazon, you always want to be able to sell, but if you’re a bank, you probably don’t want your clients to have negative numbers in their account!&lt;/p&gt;
2317&lt;p&gt;Regardless of what we do, in a distributed system, the CAP theorem always applies: Consistecy, Availability, Partitioning-tolerancy (error tolerancy). It is &lt;strong&gt;impossible&lt;/strong&gt; to guarantee all 3 at 100%. Most of the times, it does work, but it is mathematically impossible to guarantee at 100%.&lt;/p&gt;
2318&lt;p&gt;A database has to choose what to give up at some point. When designing a distributed system, this must be considered. Normally, the choice is made between consistency or response time.&lt;/p&gt;
2319&lt;h2 class=&quot;title&quot; id=&quot;further_reading&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#further_reading&quot;&gt;¶&lt;/a&gt;Further reading&lt;/h2&gt;
2320&lt;ul&gt;
2321&lt;li&gt;&lt;a href=&quot;https://www.martinfowler.com/articles/nosql-intro-original.pdf&quot;&gt;The future is: &lt;del&gt;NoSQL Databases&lt;/del&gt; Polyglot Persistence&lt;/a&gt;&lt;/li&gt;
2322&lt;li&gt;&lt;a href=&quot;https://www.thoughtworks.com/insights/blog/nosql-databases-overview&quot;&gt;NoSQL Databases: An Overview&lt;/a&gt;&lt;/li&gt;
2323&lt;/ul&gt;
2324&lt;/main&gt;
2325&lt;/body&gt;
2326&lt;/html&gt;
2327 </content></entry><entry><title>Build your own PC</title><id>dist/build-your-own-pc/index.html</id><updated>2020-03-17T23:00:00+00:00</updated><published>2020-02-24T23:00:00+00:00</published><summary>…where PC obviously stands for Personal Crawler</summary><content type="html" src="dist/build-your-own-pc/index.html">&lt;!DOCTYPE html&gt;
2328&lt;html&gt;
2329&lt;head&gt;
2330&lt;meta charset=&quot;utf-8&quot; /&gt;
2331&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
2332&lt;title&gt;Build your own PC&lt;/title&gt;
2333&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
2334&lt;/head&gt;
2335&lt;body&gt;
2336&lt;main&gt;
2337&lt;p&gt;&lt;em&gt;…where PC obviously stands for Personal Crawler&lt;/em&gt;.&lt;/p&gt;
2338&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-02-25&lt;br&gt;
2339Modified 2020-03-18&lt;/div&gt;
2340&lt;hr /&gt;
2341&lt;p&gt;This post contains the source code for a very simple crawler written in Java. You can compile and run it on any file or directory, and it will calculate the frequency of all the words it finds.&lt;/p&gt;
2342&lt;h2 class=&quot;title&quot; id=&quot;source_code&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#source_code&quot;&gt;¶&lt;/a&gt;Source code&lt;/h2&gt;
2343&lt;p&gt;Paste the following code in a new file called &lt;code&gt;Crawl.java&lt;/code&gt;:&lt;/p&gt;
2344&lt;pre&gt;&lt;code&gt;import java.io.*;
2345import java.util.*;
2346import java.util.regex.Matcher;
2347import java.util.regex.Pattern;
2348
2349class Crawl {
2350	// Regex used to tokenize the words from a line of text
2351	private final static Pattern WORDS = Pattern.compile(&amp;quot;\\w+&amp;quot;);
2352
2353	// The file where we will cache our results
2354	private final static File INDEX_FILE = new File(&amp;quot;index.bin&amp;quot;);
2355
2356	// Helper method to determine if a file is a text file or not
2357	private static boolean isTextFile(File file) {
2358		String name = file.getName().toLowerCase();
2359		return name.endsWith(&amp;quot;.txt&amp;quot;)
2360				|| name.endsWith(&amp;quot;.java&amp;quot;)
2361				|| name.endsWith(&amp;quot;.c&amp;quot;)
2362				|| name.endsWith(&amp;quot;.cpp&amp;quot;)
2363				|| name.endsWith(&amp;quot;.h&amp;quot;)
2364				|| name.endsWith(&amp;quot;.hpp&amp;quot;)
2365				|| name.endsWith(&amp;quot;.html&amp;quot;)
2366				|| name.endsWith(&amp;quot;.css&amp;quot;)
2367				|| name.endsWith(&amp;quot;.js&amp;quot;);
2368	}
2369
2370	// Normalizes a string by converting it to lowercase and removing accents
2371	private static String normalize(String string) {
2372		return string.toLowerCase()
2373				.replace(&amp;quot;á&amp;quot;, &amp;quot;a&amp;quot;)
2374				.replace(&amp;quot;é&amp;quot;, &amp;quot;e&amp;quot;)
2375				.replace(&amp;quot;í&amp;quot;, &amp;quot;i&amp;quot;)
2376				.replace(&amp;quot;ó&amp;quot;, &amp;quot;o&amp;quot;)
2377				.replace(&amp;quot;ú&amp;quot;, &amp;quot;u&amp;quot;);
2378	}
2379
2380	// Recursively fills the map with the count of words found on all the text files
2381	static void fillWordMap(Map&amp;lt;String, Integer&amp;gt; map, File root) throws IOException {
2382		// Our file queue begins with the root
2383		Queue&amp;lt;File&amp;gt; fileQueue = new ArrayDeque&amp;lt;&amp;gt;();
2384		fileQueue.add(root);
2385
2386		// For as long as the queue is not empty...
2387		File file;
2388		while ((file = fileQueue.poll()) != null) {
2389			if (!file.exists() || !file.canRead()) {
2390				// ...ignore files for which we don't have permission...
2391				System.err.println(&amp;quot;warning: cannot read file: &amp;quot; + file);
2392			} else if (file.isDirectory()) {
2393				// ...else if it's a directory, extend our queue with its files...
2394				File[] files = file.listFiles();
2395				if (files == null) {
2396					System.err.println(&amp;quot;warning: cannot list dir: &amp;quot; + file);
2397				} else {
2398					fileQueue.addAll(Arrays.asList(files));
2399				}
2400			} else if (isTextFile(file)) {
2401				// ...otherwise, count the words in the file.
2402				countWordsInFile(map, file);
2403			}
2404		}
2405	}
2406
2407	// Counts the words in a single file and adds the count to the map.
2408	public static void countWordsInFile(Map&amp;lt;String, Integer&amp;gt; map, File file) throws IOException {
2409		BufferedReader reader = new BufferedReader(new FileReader(file));
2410
2411		String line;
2412		while ((line = reader.readLine()) != null) {
2413			Matcher matcher = WORDS.matcher(line);
2414			while (matcher.find()) {
2415				String token = normalize(matcher.group());
2416				Integer count = map.get(token);
2417				if (count == null) {
2418					map.put(token, 1);
2419				} else {
2420					map.put(token, count + 1);
2421				}
2422			}
2423		}
2424
2425		reader.close();
2426	}
2427
2428	// Prints the map of word count to the desired output stream.
2429	public static void printWordMap(Map&amp;lt;String, Integer&amp;gt; map, PrintStream writer) {
2430		List&amp;lt;String&amp;gt; keys = new ArrayList&amp;lt;&amp;gt;(map.keySet());
2431		Collections.sort(keys);
2432		for (String key : keys) {
2433			writer.println(key + &amp;quot;\t&amp;quot; + map.get(key));
2434		}
2435	}
2436
2437	@SuppressWarnings(&amp;quot;unchecked&amp;quot;)
2438	public static void main(String[] args) throws IOException, ClassNotFoundException {
2439		// Validate arguments
2440		if (args.length == 1 &amp;amp;&amp;amp; args[0].equals(&amp;quot;--help&amp;quot;)) {
2441			System.err.println(&amp;quot;usage: java Crawl [input]&amp;quot;);
2442			return;
2443		}
2444
2445		File root = new File(args.length &amp;gt; 0 ? args[0] : &amp;quot;.&amp;quot;);
2446
2447		// Loading or generating the map where we aggregate the data  {word: count}
2448		Map&amp;lt;String, Integer&amp;gt; map;
2449		if (INDEX_FILE.isFile()) {
2450			System.err.println(&amp;quot;Found existing index file: &amp;quot; + INDEX_FILE);
2451			try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(INDEX_FILE))) {
2452				map = (Map&amp;lt;String, Integer&amp;gt;) ois.readObject();
2453			}
2454		} else {
2455			System.err.println(&amp;quot;Index file not found: &amp;quot; + INDEX_FILE + &amp;quot;; indexing...&amp;quot;);
2456			map = new TreeMap&amp;lt;&amp;gt;();
2457			fillWordMap(map, root);
2458			// Cache the results to avoid doing the work a next time
2459			try (ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(INDEX_FILE))) {
2460				out.writeObject(map);
2461			}
2462		}
2463
2464		// Ask the user in a loop to query for words
2465		Scanner scanner = new Scanner(System.in);
2466		while (true) {
2467			System.out.print(&amp;quot;Escriba palabra a consultar (o Enter para salir): &amp;quot;);
2468			System.out.flush();
2469			String line = scanner.nextLine().trim();
2470			if (line.isEmpty()) {
2471				break;
2472			}
2473
2474			line = normalize(line);
2475			Integer count = map.get(line);
2476			if (count == null) {
2477				System.out.println(String.format(&amp;quot;La palabra \&amp;quot;%s\&amp;quot; no está presente&amp;quot;, line));
2478			} else if (count == 1) {
2479				System.out.println(String.format(&amp;quot;La palabra \&amp;quot;%s\&amp;quot; está presente 1 vez&amp;quot;, line));
2480			} else {
2481				System.out.println(String.format(&amp;quot;La palabra \&amp;quot;%s\&amp;quot; está presente %d veces&amp;quot;, line, count));
2482			}
2483		}
2484	}
2485}
2486&lt;/code&gt;&lt;/pre&gt;
2487&lt;p&gt;It can be compiled and executed as follows:&lt;/p&gt;
2488&lt;pre&gt;&lt;code&gt;javac Crawl.java
2489java Crawl
2490&lt;/code&gt;&lt;/pre&gt;
2491&lt;p&gt;Instead of copy-pasting the code, you may also download it as a &lt;code&gt;.zip&lt;/code&gt;:&lt;/p&gt;
2492&lt;p&gt;&lt;em&gt;(contents removed)&lt;/em&gt;&lt;/p&gt;
2493&lt;h2 id=&quot;addendum&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#addendum&quot;&gt;¶&lt;/a&gt;Addendum&lt;/h2&gt;
2494&lt;p&gt;The following simple function can be used if one desires to print the contents of a file:&lt;/p&gt;
2495&lt;pre&gt;&lt;code&gt;public static void printFile(File file) {
2496	if (isTextFile(file)) {
2497		System.out.println('\n' + file.getName());
2498		try (BufferedReader reader = new BufferedReader(new FileReader(file))) {
2499			String line;
2500			while ((line = reader.readLine()) != null) {
2501				System.out.println(line);
2502			}
2503		} catch (FileNotFoundException ignored) {
2504			System.err.println(&amp;quot;warning: file disappeared while reading: &amp;quot; + file);
2505		} catch (IOException e) {
2506			e.printStackTrace();
2507		}
2508	}
2509}
2510&lt;/code&gt;&lt;/pre&gt;
2511&lt;/main&gt;
2512&lt;/body&gt;
2513&lt;/html&gt;
2514 </content></entry><entry><title>About Boolean Retrieval</title><id>dist/about-boolean-retrieval/index.html</id><updated>2020-03-17T23:00:00+00:00</updated><published>2020-02-24T23:00:00+00:00</published><summary>This entry will discuss the section on the </summary><content type="html" src="dist/about-boolean-retrieval/index.html">&lt;!DOCTYPE html&gt;
2515&lt;html&gt;
2516&lt;head&gt;
2517&lt;meta charset=&quot;utf-8&quot; /&gt;
2518&lt;meta name=&quot;viewport&quot; content=&quot;width=device-width, initial-scale=1&quot; /&gt;
2519&lt;title&gt;About Boolean Retrieval&lt;/title&gt;
2520&lt;link rel=&quot;stylesheet&quot; href=&quot;../css/style.css&quot;&gt;
2521&lt;/head&gt;
2522&lt;body&gt;
2523&lt;main&gt;
2524&lt;p&gt;This entry will discuss the section on the &lt;em&gt;&lt;a href=&quot;https://nlp.stanford.edu/IR-book/pdf/01bool.pdf&quot;&gt;Boolean retrieval&lt;/a&gt;&lt;/em&gt; section of the book &lt;em&gt;&lt;a href=&quot;https://nlp.stanford.edu/IR-book/pdf/irbookprint.pdf&quot;&gt;An Introduction to Information Retrieval&lt;/a&gt;&lt;/em&gt;.&lt;/p&gt;
2525&lt;div class=&quot;date-created-modified&quot;&gt;Created 2020-02-25&lt;br&gt;
2526Modified 2020-03-18&lt;/div&gt;
2527&lt;h2 class=&quot;title&quot; id=&quot;summary_on_the_topic&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#summary_on_the_topic&quot;&gt;¶&lt;/a&gt;Summary on the topic&lt;/h2&gt;
2528&lt;p&gt;Boolean retrieval is one of the many ways information retrieval (finding materials that satisfy an information need), often simply called &lt;em&gt;search&lt;/em&gt;.&lt;/p&gt;
2529&lt;p&gt;A simple way to retrieve information is to &lt;em&gt;grep&lt;/em&gt; through the text (term named after the Unix tool &lt;code&gt;grep&lt;/code&gt;), scanning text linearly and excluding it on certain criteria. However, this falls short when the volume of the data grows, more complex queries are desired, or one seeks some sort of ranking.&lt;/p&gt;
2530&lt;p&gt;To avoid linear scanning, we build an &lt;em&gt;index&lt;/em&gt; and record for each document whether it contains each term out of our full dictionary of terms (which may be words in a chapter and words in the book). This results in a binary term-document &lt;em&gt;incidence matrix&lt;/em&gt;. Such a possible matrix is:&lt;/p&gt;
2531&lt;table class=&quot;&quot;&gt;
2532 &lt;tbody&gt;
2533  &lt;tr&gt;
2534   &lt;td&gt;
2535    &lt;em&gt;
2536     word/play
2537    &lt;/em&gt;
2538   &lt;/td&gt;
2539   &lt;td&gt;
2540    &lt;strong&gt;
2541     Antony and Cleopatra
2542    &lt;/strong&gt;
2543   &lt;/td&gt;
2544   &lt;td&gt;
2545    &lt;strong&gt;
2546     Julius Caesar
2547    &lt;/strong&gt;
2548   &lt;/td&gt;
2549   &lt;td&gt;
2550    &lt;strong&gt;
2551     The Tempest
2552    &lt;/strong&gt;
2553   &lt;/td&gt;
2554   &lt;td&gt;
2555    &lt;strong&gt;
2556     …
2557    &lt;/strong&gt;
2558   &lt;/td&gt;
2559  &lt;/tr&gt;
2560  &lt;tr&gt;
2561   &lt;td&gt;
2562    &lt;strong&gt;
2563     Antony
2564    &lt;/strong&gt;
2565   &lt;/td&gt;
2566   &lt;td&gt;
2567    1
2568   &lt;/td&gt;
2569   &lt;td&gt;
2570    1
2571   &lt;/td&gt;
2572   &lt;td&gt;
2573    0
2574   &lt;/td&gt;
2575   &lt;td&gt;
2576   &lt;/td&gt;
2577  &lt;/tr&gt;
2578  &lt;tr&gt;
2579   &lt;td&gt;
2580    &lt;strong&gt;
2581     Brutus
2582    &lt;/strong&gt;
2583   &lt;/td&gt;
2584   &lt;td&gt;
2585    1
2586   &lt;/td&gt;
2587   &lt;td&gt;
2588    1
2589   &lt;/td&gt;
2590   &lt;td&gt;
2591    0
2592   &lt;/td&gt;
2593   &lt;td&gt;
2594   &lt;/td&gt;
2595  &lt;/tr&gt;
2596  &lt;tr&gt;
2597   &lt;td&gt;
2598    &lt;strong&gt;
2599     Caesar
2600    &lt;/strong&gt;
2601   &lt;/td&gt;
2602   &lt;td&gt;
2603    1
2604   &lt;/td&gt;
2605   &lt;td&gt;
2606    1
2607   &lt;/td&gt;
2608   &lt;td&gt;
2609    0
2610   &lt;/td&gt;
2611   &lt;td&gt;
2612   &lt;/td&gt;
2613  &lt;/tr&gt;
2614  &lt;tr&gt;
2615   &lt;td&gt;
2616    &lt;strong&gt;
2617     Calpurnia
2618    &lt;/strong&gt;
2619   &lt;/td&gt;
2620   &lt;td&gt;
2621    0
2622   &lt;/td&gt;
2623   &lt;td&gt;
2624    1
2625   &lt;/td&gt;
2626   &lt;td&gt;
2627    0
2628   &lt;/td&gt;
2629   &lt;td&gt;
2630   &lt;/td&gt;
2631  &lt;/tr&gt;
2632  &lt;tr&gt;
2633   &lt;td&gt;
2634    &lt;strong&gt;
2635     Cleopatra
2636    &lt;/strong&gt;
2637   &lt;/td&gt;
2638   &lt;td&gt;
2639    1
2640   &lt;/td&gt;
2641   &lt;td&gt;
2642    0
2643   &lt;/td&gt;
2644   &lt;td&gt;
2645    0
2646   &lt;/td&gt;
2647   &lt;td&gt;
2648   &lt;/td&gt;
2649  &lt;/tr&gt;
2650  &lt;tr&gt;
2651   &lt;td&gt;
2652    &lt;strong&gt;
2653     mercy
2654    &lt;/strong&gt;
2655   &lt;/td&gt;
2656   &lt;td&gt;
2657    1
2658   &lt;/td&gt;
2659   &lt;td&gt;
2660    0
2661   &lt;/td&gt;
2662   &lt;td&gt;
2663    1
2664   &lt;/td&gt;
2665   &lt;td&gt;
2666   &lt;/td&gt;
2667  &lt;/tr&gt;
2668  &lt;tr&gt;
2669   &lt;td&gt;
2670    &lt;strong&gt;
2671     worser
2672    &lt;/strong&gt;
2673   &lt;/td&gt;
2674   &lt;td&gt;
2675    1
2676   &lt;/td&gt;
2677   &lt;td&gt;
2678    0
2679   &lt;/td&gt;
2680   &lt;td&gt;
2681    1
2682   &lt;/td&gt;
2683   &lt;td&gt;
2684   &lt;/td&gt;
2685  &lt;/tr&gt;
2686  &lt;tr&gt;
2687   &lt;td&gt;
2688    &lt;strong&gt;
2689     …
2690    &lt;/strong&gt;
2691   &lt;/td&gt;
2692   &lt;td&gt;
2693   &lt;/td&gt;
2694   &lt;td&gt;
2695   &lt;/td&gt;
2696   &lt;td&gt;
2697   &lt;/td&gt;
2698   &lt;td&gt;
2699   &lt;/td&gt;
2700  &lt;/tr&gt;
2701 &lt;/tbody&gt;
2702&lt;/table&gt;
2703&lt;p&gt;We can look at this matrix’s rows or columns to obtain a vector for each term indicating where it appears, or a vector for each document indicating the terms it contains.&lt;/p&gt;
2704&lt;p&gt;Now, answering a query such as &lt;code&gt;Brutus AND Caesar AND NOT Calpurnia&lt;/code&gt; becomes trivial:&lt;/p&gt;
2705&lt;pre&gt;&lt;code&gt;VECTOR(Brutus) AND VECTOR(Caesar) AND COMPLEMENT(VECTOR(Calpurnia))
2706= 110 AND 110 AND COMPLEMENT(010)
2707= 110 AND 110 AND 101
2708= 100
2709&lt;/code&gt;&lt;/pre&gt;
2710&lt;p&gt;The query is only satisfied for our first column.&lt;/p&gt;
2711&lt;p&gt;The &lt;em&gt;Boolean retrieval model&lt;/em&gt; is thus a model that treats documents as a set of terms, in which we can perform any query in the form of Boolean expressions of terms, combined with &lt;code&gt;OR&lt;/code&gt;, &lt;code&gt;AND&lt;/code&gt;, and &lt;code&gt;NOT&lt;/code&gt;.&lt;/p&gt;
2712&lt;p&gt;Now, building such a matrix is often not feasible due to the sheer amount of data (say, a matrix with 500,000 terms across 1,000,000 documents, each with roughly 1,000 terms). However, it is important to notice that most of the terms will be &lt;em&gt;missing&lt;/em&gt; when examining each document. In our example, this means 99.8% or more of the cells will be 0. We can instead record the &lt;em&gt;positions&lt;/em&gt; of the 1’s. This is known as an &lt;em&gt;inverted index&lt;/em&gt;.&lt;/p&gt;
2713&lt;p&gt;The inverted index is a dictionary of terms, each containing a list that records in which documents it appears (&lt;em&gt;postings&lt;/em&gt;). Applied to boolean retrieval, we would:&lt;/p&gt;
2714&lt;ol&gt;
2715&lt;li&gt;Collects the documents to be indexed, assign a unique identifier each&lt;/li&gt;
2716&lt;li&gt;Tokenize the text in the documents into a list of terms&lt;/li&gt;
2717&lt;li&gt;Normalize the tokens, which now become indexing terms&lt;/li&gt;
2718&lt;li&gt;Index the documents&lt;/li&gt;
2719&lt;/ol&gt;
2720&lt;table class=&quot;&quot;&gt;
2721 &lt;tbody&gt;
2722  &lt;tr&gt;
2723   &lt;td&gt;
2724    &lt;strong&gt;
2725     Dictionary
2726    &lt;/strong&gt;
2727   &lt;/td&gt;
2728   &lt;td&gt;
2729    &lt;strong&gt;
2730     Postings
2731    &lt;/strong&gt;
2732   &lt;/td&gt;
2733  &lt;/tr&gt;
2734  &lt;tr&gt;
2735   &lt;td&gt;
2736    Brutus
2737   &lt;/td&gt;
2738   &lt;td&gt;
2739    1, 2, 4, 11, 31, 45, 173, 174
2740   &lt;/td&gt;
2741  &lt;/tr&gt;
2742  &lt;tr&gt;
2743   &lt;td&gt;
2744    Caesar
2745   &lt;/td&gt;
2746   &lt;td&gt;
2747    1, 2, 4, 5, 6, 16, 57, 132, …
2748   &lt;/td&gt;
2749  &lt;/tr&gt;
2750  &lt;tr&gt;
2751   &lt;td&gt;
2752    Calpurnia
2753   &lt;/td&gt;
2754   &lt;td&gt;
2755    2, 31, 54, 101
2756   &lt;/td&gt;
2757  &lt;/tr&gt;
2758  &lt;tr&gt;
2759   &lt;td&gt;
2760    …
2761   &lt;/td&gt;
2762   &lt;td&gt;
2763   &lt;/td&gt;
2764  &lt;/tr&gt;
2765 &lt;/tbody&gt;
2766&lt;/table&gt;
2767&lt;p&gt;Sort the pairs &lt;code&gt;(term, document_id)&lt;/code&gt; so that the terms are alphabetical, and merge multiple occurences into one. Group instances of the same term and split again into a sorted list of postings.&lt;/p&gt;
2768&lt;table class=&quot;&quot;&gt;
2769 &lt;tbody&gt;
2770  &lt;tr&gt;
2771   &lt;td&gt;
2772    &lt;strong&gt;
2773     term
2774    &lt;/strong&gt;
2775   &lt;/td&gt;
2776   &lt;td&gt;
2777    &lt;strong&gt;
2778     document_id
2779    &lt;/strong&gt;
2780   &lt;/td&gt;
2781  &lt;/tr&gt;
2782  &lt;tr&gt;
2783   &lt;td&gt;
2784    I
2785   &lt;/td&gt;
2786   &lt;td&gt;
2787    1
2788   &lt;/td&gt;
2789  &lt;/tr&gt;
2790  &lt;tr&gt;
2791   &lt;td&gt;
2792    did
2793   &lt;/td&gt;
2794   &lt;td&gt;
2795    1
2796   &lt;/td&gt;
2797  &lt;/tr&gt;
2798  &lt;tr&gt;
2799   &lt;td&gt;
2800    …
2801   &lt;/td&gt;
2802   &lt;td&gt;
2803   &lt;/td&gt;
2804  &lt;/tr&gt;
2805  &lt;tr&gt;
2806   &lt;td&gt;
2807    with
2808   &lt;/td&gt;
2809   &lt;td&gt;
2810    2
2811   &lt;/td&gt;
2812  &lt;/tr&gt;
2813 &lt;/tbody&gt;
2814&lt;/table&gt;
2815&lt;table class=&quot;&quot;&gt;
2816 &lt;tbody&gt;
2817  &lt;tr&gt;
2818   &lt;td&gt;
2819    &lt;strong&gt;
2820     term
2821    &lt;/strong&gt;
2822   &lt;/td&gt;
2823   &lt;td&gt;
2824    &lt;strong&gt;
2825     document_id
2826    &lt;/strong&gt;
2827   &lt;/td&gt;
2828  &lt;/tr&gt;
2829  &lt;tr&gt;
2830   &lt;td&gt;
2831    be
2832   &lt;/td&gt;
2833   &lt;td&gt;
2834    2
2835   &lt;/td&gt;
2836  &lt;/tr&gt;
2837  &lt;tr&gt;
2838   &lt;td&gt;
2839    brutus
2840   &lt;/td&gt;
2841   &lt;td&gt;
2842    1
2843   &lt;/td&gt;
2844  &lt;/tr&gt;
2845  &lt;tr&gt;
2846   &lt;td&gt;
2847    brutus
2848   &lt;/td&gt;
2849   &lt;td&gt;
2850    2
2851   &lt;/td&gt;
2852  &lt;/tr&gt;
2853  &lt;tr&gt;
2854   &lt;td&gt;
2855    …
2856   &lt;/td&gt;
2857   &lt;td&gt;
2858   &lt;/td&gt;
2859  &lt;/tr&gt;
2860 &lt;/tbody&gt;
2861&lt;/table&gt;
2862&lt;table class=&quot;&quot;&gt;
2863 &lt;tbody&gt;
2864  &lt;tr&gt;
2865   &lt;td&gt;
2866    &lt;strong&gt;
2867     term
2868    &lt;/strong&gt;
2869   &lt;/td&gt;
2870   &lt;td&gt;
2871    &lt;strong&gt;
2872     frequency
2873    &lt;/strong&gt;
2874   &lt;/td&gt;
2875   &lt;td&gt;
2876    &lt;strong&gt;
2877     postings list
2878    &lt;/strong&gt;
2879   &lt;/td&gt;
2880  &lt;/tr&gt;
2881  &lt;tr&gt;
2882   &lt;td&gt;
2883    be
2884   &lt;/td&gt;
2885   &lt;td&gt;
2886    1
2887   &lt;/td&gt;
2888   &lt;td&gt;
2889    2
2890   &lt;/td&gt;
2891  &lt;/tr&gt;
2892  &lt;tr&gt;
2893   &lt;td&gt;
2894    brutus
2895   &lt;/td&gt;
2896   &lt;td&gt;
2897    2
2898   &lt;/td&gt;
2899   &lt;td&gt;
2900    1, 2
2901   &lt;/td&gt;
2902  &lt;/tr&gt;
2903  &lt;tr&gt;
2904   &lt;td&gt;
2905    capitol
2906   &lt;/td&gt;
2907   &lt;td&gt;
2908    1
2909   &lt;/td&gt;
2910   &lt;td&gt;
2911    1
2912   &lt;/td&gt;
2913  &lt;/tr&gt;
2914  &lt;tr&gt;
2915   &lt;td&gt;
2916    …
2917   &lt;/td&gt;
2918   &lt;td&gt;
2919   &lt;/td&gt;
2920   &lt;td&gt;
2921   &lt;/td&gt;
2922  &lt;/tr&gt;
2923 &lt;/tbody&gt;
2924&lt;/table&gt;
2925&lt;p&gt;Intersecting posting lists now becomes of transversing both lists in order:&lt;/p&gt;
2926&lt;pre&gt;&lt;code&gt;Brutus   : 1 -&amp;gt; 2 -&amp;gt; 4 -&amp;gt; 11 -&amp;gt; 31 -&amp;gt; 45 -&amp;gt;           173 -&amp;gt; 174
2927Calpurnia:      2 -&amp;gt;            31 -&amp;gt;       54 -&amp;gt; 101
2928Intersect:      2 -&amp;gt;            31
2929&lt;/code&gt;&lt;/pre&gt;
2930&lt;p&gt;A simple conjunctive query (e.g. &lt;code&gt;Brutus AND Calpurnia&lt;/code&gt;) is executed as follows:&lt;/p&gt;
2931&lt;ol&gt;
2932&lt;li&gt;Locate &lt;code&gt;Brutus&lt;/code&gt; in the dictionary&lt;/li&gt;
2933&lt;li&gt;Retrieve its postings&lt;/li&gt;
2934&lt;li&gt;Locate &lt;code&gt;Calpurnia&lt;/code&gt; in the dictionary&lt;/li&gt;
2935&lt;li&gt;Retrieve its postings&lt;/li&gt;
2936&lt;li&gt;Intersect (&lt;em&gt;merge&lt;/em&gt;) both postings&lt;/li&gt;
2937&lt;/ol&gt;
2938&lt;p&gt;Since the lists are sorted, walking both of them can be done in &lt;em&gt;O(n)&lt;/em&gt; time. By also storing the frequency, we can optimize the order in which we execute arbitrary queries, although we won’t go into detail.&lt;/p&gt;
2939&lt;h2 id=&quot;thoughts&quot;&gt;&lt;a class=&quot;anchor&quot; href=&quot;#thoughts&quot;&gt;¶&lt;/a&gt;Thoughts&lt;/h2&gt;
2940&lt;p&gt;The boolean retrieval model can be implemented with relative ease, and can help with storage and efficient querying of the information if we intend to perform boolean queries.&lt;/p&gt;
2941&lt;p&gt;However, the basic design lacks other useful operations, such as a «near» operator, or the ability to rank the results.&lt;/p&gt;
2942&lt;p&gt;All in all, it’s an interesting way to look at the data and query it efficiently.&lt;/p&gt;
2943&lt;/main&gt;
2944&lt;/body&gt;
2945&lt;/html&gt;
2946 </content></entry></feed>