blog/ribw/googles-bigtable/index.html (view raw)
1<!DOCTYPE html>
2<html>
3<head>
4<meta charset="utf-8" />
5<meta name="viewport" content="width=device-width, initial-scale=1" />
6<title>Google’s BigTable</title>
7<link rel="stylesheet" href="../css/style.css">
8</head>
9<body>
10<main>
11<p>Let’s talk about BigTable, and why it is what it is. But before we get into that, let’s see some important aspects anybody should consider when dealing with a lot of data (something BigTable does!).</p>
12<div class="date-created-modified">Created 2020-04-01<br>
13Modified 2020-04-03</div>
14<h2 class="title" id="the_basics"><a class="anchor" href="#the_basics">¶</a>The basics</h2>
15<p>Converting a text document into a different format is often a great way to greatly speed up scanning of it in the future. It allows for efficient searches.</p>
16<p>In addition, you generally want to store everything in a single, giant file. This will save a lot of time opening and closing files, because everything is in the same file! One proposal to make this happen is <a href="https://trec.nist.gov/file_help.html">Web TREC</a> (see also the <a href="https://en.wikipedia.org/wiki/Text_Retrieval_Conference">Wikipedia page on TREC</a>), which is basically HTML but every document is properly delimited from one another.</p>
17<p>Because we will have a lot of data, it’s often a good idea to compress it. Most text consists of the same words, over and over again. Classic compression techniques such as <code>DEFLATE</code> or <code>LZW</code> do an excellent job here.</p>
18<h2 id="so_what_s_bigtable_"><a class="anchor" href="#so_what_s_bigtable_">¶</a>So what’s BigTable?</h2>
19<p>Okay, enough of an introduction to the basics on storing data. BigTable is what Google uses to store documents, and it’s a customized approach to save, search and update web pages.</p>
20<p>BigTable is is a distributed storage system for managing structured data, able to scale to petabytes of data across thousands of commodity servers, with wide applicability, scalability, high performance, and high availability.</p>
21<p>In a way, it’s kind of like databases and shares many implementation strategies with them, like parallel databases, or main-memory databases, but of course, with a different schema.</p>
22<p>It consists of a big table known as the «Root tablet», with pointers to many other «tablets» (or metadata in between). These are stored in a replicated filesystem accessible by all BigTable servers. Any change to a tablet gets logged (said log also gets stored in a replicated filesystem).</p>
23<p>If any of the tablets servers gets locked, a different one can take its place, read the log and deal with the problem.</p>
24<p>There’s no query language, transactions occur at row-level only. Every read or write in a row is atomic. Each row stores a single web page, and by combining the row and column keys along with a timestamp, it is possible to retrieve a single cell in the row. More formally, it’s a map that looks like this:</p>
25<pre><code>fetch(row: string, column: string, time: int64) -> string
26</code></pre>
27<p>A row may have as many columns as it needs, and these column groups are the same for everyone (but the columns themselves may vary), which is importan to reduce disk read time.</p>
28<p>Rows are split in different tablets based on the row keys, which simplifies determining an appropriated server for them. The keys can be up to 64KB big, although most commonly they range 10-100 bytes.</p>
29<h2 id="conclusions"><a class="anchor" href="#conclusions">¶</a>Conclusions</h2>
30<p>BigTable is Google’s way to deal with large amounts of data on many of their services, and the ideas behind it are not too complex to understand.</p>
31</main>
32</body>
33</html>
34