blog/ribw/developing-a-python-application-for-mongodb/index.html (view raw)
1<!DOCTYPE html>
2<html>
3<head>
4<meta charset="utf-8" />
5<meta name="viewport" content="width=device-width, initial-scale=1" />
6<title>Developing a Python application for MongoDB</title>
7<link rel="stylesheet" href="../css/style.css">
8</head>
9<body>
10<main>
11<p>This is the third and last post in the MongoDB series, where we will develop a Python application to process and store OpenData inside Mongo.</p>
12<div class="date-created-modified">Created 2020-03-25<br>
13Modified 2020-04-16</div>
14<p>Other posts in this series:</p>
15<ul>
16<li><a href="/blog/ribw/mongodb-an-introduction/">MongoDB: an Introduction</a></li>
17<li><a href="/blog/ribw/mongodb-basic-operations-and-architecture/">MongoDB: Basic Operations and Architecture</a></li>
18<li><a href="/blog/ribw/developing-a-python-application-for-mongodb/">Developing a Python application for MongoDB</a> (this post)</li>
19</ul>
20<p>This post is co-authored wih a Classmate.</p>
21<hr />
22<h2 class="title" id="what_are_we_making_"><a class="anchor" href="#what_are_we_making_">¶</a>What are we making?</h2>
23<p>We are going to develop a web application that renders a map, in this case, the town of Cáceres, with which users can interact. When the user clicks somewhere on the map, the selected location will be sent to the server to process. This server will perform geospatial queries to Mongo and once the results are ready, the information is presented back at the webpage.</p>
24<p>The data used for the application comes from <a href="https://opendata.caceres.es/">Cáceres’ OpenData</a>, and our goal is that users will be able to find information about certain areas in a quick and intuitive way, such as precise coordinates, noise level, and such.</p>
25<h2 id="what_are_we_using_"><a class="anchor" href="#what_are_we_using_">¶</a>What are we using?</h2>
26<p>The web application will be using <a href="https://python.org/">Python</a> for the backend, <a href="https://svelte.dev/">Svelte</a> for the frontend, and <a href="https://www.mongodb.com/">Mongo</a> as our storage database and processing center.</p>
27<ul>
28<li><strong>Why Python?</strong> It’s a comfortable language to write and to read, and has a great ecosystem with <a href="https://pypi.org/">plenty of libraries</a>.</li>
29<li><strong>Why Svelte?</strong> Svelte is the New Thing<strong>™</strong> in the world of component frameworks for JavaScript. It is similar to React or Vue, but compiled and with a lot less boilerplate. Check out their <a href="https://svelte.dev/blog/svelte-3-rethinking-reactivity">Svelte post</a> to learn more.</li>
30<li><strong>Why Mongo?</strong> We believe NoSQL is the right approach for doing the kind of processing and storage that we expect, and it’s <a href="https://docs.mongodb.com/">very easy to use</a>. In addition, we will be making Geospatial Queries which <a href="https://docs.mongodb.com/manual/geospatial-queries/">Mongo supports</a>.</li>
31</ul>
32<p>Why didn’t we choose to make a smaller project, you may ask? You will be shocked to hear that we do not have an answer for that!</p>
33<p>Note that we will not be embedding <strong>all</strong> the code of the project in this post, or it would be too long! We will include only the relevant snippets needed to understand the core ideas of the project, and not the unnecessary parts of it (for example, parsing configuration files to easily change the port where the server runs is not included).</p>
34<h2 id="python_dependencies"><a class="anchor" href="#python_dependencies">¶</a>Python dependencies</h2>
35<p>Because we will program it in Python, you need Python installed. You can install it using a package manager of your choice or heading over to the <a href="https://www.python.org/downloads/">Python downloads section</a>, but if you’re on Linux, chances are you have it installed already.</p>
36<p>Once Python 3.7 or above is installed, install <a href="https://motor.readthedocs.io/en/stable/"><code>motor</code> (Asynchronous Python driver for MongoDB)</a> and the <a href="https://docs.aiohttp.org/en/stable/web.html"><code>aiohttp</code> server</a> through <code>pip</code>:</p>
37<pre><code>pip install aiohttp motor
38</code></pre>
39<p>Make sure that Mongo is running in the background (this has been described in previous posts), and we should be able to get to work.</p>
40<h2 id="web_dependencies"><a class="anchor" href="#web_dependencies">¶</a>Web dependencies</h2>
41<p>To work with Svelte and its dependencies, we will need <code>[npm](https://www.npmjs.com/)</code> which comes with <a href="https://nodejs.org/en/">NodeJS</a>, so go and <a href="https://nodejs.org/en/download/">install Node from their site</a>. The download will be different depending on your operating system.</p>
42<p>Following <a href="https://svelte.dev/blog/the-easiest-way-to-get-started">the easiest way to get started with Svelte</a>, we will put our project in a <code>client/</code> folder (because this is what the clients see, the frontend). Feel free to tinker a bit with the configuration files to change the name and such, although this isn’t relevant for the rest of the post.</p>
43<h2 id="finding_the_data"><a class="anchor" href="#finding_the_data">¶</a>Finding the data</h2>
44<p>We are going to work with the JSON files provided by <a href="http://opendata.caceres.es/">OpenData Cáceres</a>. In particular, we want information about the noise, census, vias and trees. To save you the time from <a href="http://opendata.caceres.es/dataset">searching each of these</a>, we will automate the download with code.</p>
45<p>If you want to save the data offline or just know what data we’ll be using for other purposes though, you can right click on the following links and select «Save Link As…» with the name of the link:</p>
46<ul>
47<li><code>[noise.json](http://opendata.caceres.es/GetData/GetData?dataset=om:MedicionRuido&format=json)</code></li>
48<li><code>[census.json](http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionPadron&year=2017&format=json)</code></li>
49<li><code>[vias.json](http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionPadron&year=2017&format=json)</code></li>
50<li><code>[trees.json](http://opendata.caceres.es/GetData/GetData?dataset=om:Arbol&format=json)</code></li>
51</ul>
52<h2 id="backend"><a class="anchor" href="#backend">¶</a>Backend</h2>
53<p>It’s time to get started with some code! We will put it in a <code>server/</code> folder because it will contain the Python server, that is, the backend of our application.</p>
54<p>We are using <code>aiohttp</code> because we would like our server to be <code>async</code>. We don’t expect a lot of users at the same time, but it’s good to know our server would be well-designed for that use-case. As a bonus, it makes IO points clear in the code, which can help reason about it. The implicit synchronization between <code>await</code> is also a nice bonus.</p>
55<h3 id="saving_the_data_in_mongo"><a class="anchor" href="#saving_the_data_in_mongo">¶</a>Saving the data in Mongo</h3>
56<p>Before running the server, we must ensure that the data we need is already stored and indexed in Mongo. Our <code>server/data.py</code> will take care of downloading the files, cleaning them up a little (Cáceres’ OpenData can be a bit awkward sometimes), inserting them into Mongo and indexing them.</p>
57<p>Downloading the JSON data can be done with <code>[ClientSession.get](https://aiohttp.readthedocs.io/en/stable/client_reference.html#aiohttp.ClientSession.get)</code>. We also take this opportunity to clean up the messy encoding from the JSON, which does not seem to be UTF-8 in some cases.</p>
58<pre><code>async def load_json(session, url):
59 fixes = [(old, new.encode('utf-8')) for old, new in [
60 (b'\xc3\x83\\u2018', 'Ñ'),
61 (b'\xc3\x83\\u0081', 'Á'),
62 (b'\xc3\x83\\u2030', 'É'),
63 (b'\xc3\x83\\u008D', 'Í'),
64 (b'\xc3\x83\\u201C', 'Ó'),
65 (b'\xc3\x83\xc5\xa1', 'Ú'),
66 (b'\xc3\x83\xc2\xa1', 'á'),
67 ]]
68
69 async with session.get(url) as resp:
70 data = await resp.read()
71
72 # Yes, this feels inefficient, but it's not really worth improving.
73 for old, new in fixes:
74 data = data.replace(old, new)
75
76 data = data.decode('utf-8')
77 return json.loads(data)
78</code></pre>
79<p>Later on, it can be reused for the various different URLs:</p>
80<pre><code>import aiohttp
81
82NOISE_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:MedicionRuido&format=json'
83# (...other needed URLs here)
84
85async def insert_to_db(db):
86 async with aiohttp.ClientSession() as session:
87 data = await load_json(session, NOISE_URL)
88 # now we have the JSON data cleaned up, ready to be parsed
89</code></pre>
90<h3 id="data_model"><a class="anchor" href="#data_model">¶</a>Data model</h3>
91<p>With the JSON data in our hands, it’s time to parse it. Always remember to <a href="https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/">parse, don’t validate</a>. With <a href="https://docs.python.org/3/library/dataclasses.html">Python 3.7 <code>dataclasses</code></a> it’s trivial to define classes that will store only the fields we care about, typed, and with proper names:</p>
92<pre><code>from dataclasses import dataclass
93
94Longitude = float
95Latitude = float
96
97@dataclass
98class GSON:
99 type: str
100 coordinates: (Longitude, Latitude)
101
102@dataclass
103class Noise:
104 id: int
105 geo: GSON
106 level: float
107</code></pre>
108<p>This makes it really easy to see that, if we have a <code>Noise</code>, we can access its <code>geo</code> data which is a <code>GSON</code> with a <code>type</code> and <code>coordinates</code>, having <code>Longitude</code> and <code>Latitude</code> respectively. <code>dataclasses</code> and <code>[typing](https://docs.python.org/3/library/typing.html)</code> make dealing with this very easy and clear.</p>
109<p>Every dataclass will be on its own collection inside Mongo, and these are:</p>
110<ul>
111<li>
112<p>Noise</p>
113</li>
114<li>
115<p>Integer <code>id</code></p>
116</li>
117<li>
118<p>GeoJSON <code>geo</code></p>
119</li>
120<li>
121<p>String <code>type</code></p>
122</li>
123<li>
124<p>Longitude-latitude pair <code>coordinates</code></p>
125</li>
126<li>
127<p>Floating-point number <code>level</code></p>
128</li>
129<li>
130<p>Tree</p>
131</li>
132<li>
133<p>String <code>name</code></p>
134</li>
135<li>
136<p>String <code>gender</code></p>
137</li>
138<li>
139<p>Integer <code>units</code></p>
140</li>
141<li>
142<p>Floating-point number <code>height</code></p>
143</li>
144<li>
145<p>Floating-point number <code>cup_diameter</code></p>
146</li>
147<li>
148<p>Floating-point number <code>trunk_diameter</code></p>
149</li>
150<li>
151<p>Optional string <code>variety</code></p>
152</li>
153<li>
154<p>Optional string <code>distribution</code></p>
155</li>
156<li>
157<p>GeoJSON <code>geo</code></p>
158</li>
159<li>
160<p>Optional string <code>irrigation</code></p>
161</li>
162<li>
163<p>Census</p>
164</li>
165<li>
166<p>Integer <code>year</code></p>
167</li>
168<li>
169<p>Via <code>via</code></p>
170</li>
171<li>
172<p>String <code>name</code></p>
173</li>
174<li>
175<p>String <code>kind</code></p>
176</li>
177<li>
178<p>Integer <code>code</code></p>
179</li>
180<li>
181<p>Optional string <code>history</code></p>
182</li>
183<li>
184<p>Optional string <code>old_name</code></p>
185</li>
186<li>
187<p>Optional floating-point number <code>length</code></p>
188</li>
189<li>
190<p>Optional GeoJSON <code>start</code></p>
191</li>
192<li>
193<p>GeoJSON <code>middle</code></p>
194</li>
195<li>
196<p>Optional GeoJSON <code>end</code></p>
197</li>
198<li>
199<p>Optional list with geometry pairs <code>geometry</code></p>
200</li>
201<li>
202<p>Integer <code>count</code></p>
203</li>
204<li>
205<p>Mapping year-to-count <code>count_per_year</code></p>
206</li>
207<li>
208<p>Mapping gender-to-count <code>count_per_gender</code></p>
209</li>
210<li>
211<p>Mapping nationality-to-count <code>count_per_nationality</code></p>
212</li>
213<li>
214<p>Integer <code>time_year</code></p>
215</li>
216</ul>
217<p>Now, let’s define a method to actually parse the JSON and yield instances from these new data classes:</p>
218<pre><code>@classmethod
219def iter_from_json(cls, data):
220 for row in data['results']['bindings']:
221 noise_id = int(row['uri']['value'].split('/')[-1])
222 long = float(row['geo_long']['value'])
223 lat = float(row['geo_lat']['value'])
224 level = float(row['om_nivelRuido']['value'])
225
226 yield cls(
227 id=noise_id,
228 geo=GSON(type='Point', coordinates=[long, lat]),
229 level=level
230 )
231</code></pre>
232<p>Here we iterate over the input JSON <code>data</code> bindings and <code>yield cls</code> instances with more consistent naming than the original one. We also extract the data from the many unnecessary nested levels of the JSON and have something a lot flatter to work with.</p>
233<p>For those of you who don’t know what <code>yield</code> does (after all, not everyone is used to seeing generators), here’s two functions that work nearly the same:</p>
234<pre><code>def squares_return(n):
235 result = []
236 for i in range(n):
237 result.append(n ** 2)
238 return result
239
240def squares_yield(n):
241 for i in range(n):
242 yield n ** 2
243</code></pre>
244<p>The difference is that the one with <code>yield</code> is «lazy» and doesn’t need to do all the work up-front. It will generate (yield) more values as they are needed when you use a <code>for</code> loop. Generally, it’s a better idea to create generator functions than do all the work early which may be unnecessary. See <a href="https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do">What does the «yield» keyword do?</a> if you still have questions.</p>
245<p>With everything parsed, it’s time to insert the data into Mongo. If the data was not present yet (0 documents), then we will download the file, parse it, insert it as documents into the given Mongo <code>db</code>, and index it:</p>
246<pre><code>from dataclasses import asdict
247
248async def insert_to_db(db):
249 async with aiohttp.ClientSession() as session:
250 if await db.noise.estimated_document_count() == 0:
251 data = await load_json(session, NOISE_URL)
252
253 await db.noise.insert_many(asdict(noise) for noise in Noise.iter_from_json(data))
254 await db.noise.create_index([('geo', '2dsphere')])
255</code></pre>
256<p>We repeat this process for all the other data, and just like that, Mongo is ready to be used in our server.</p>
257<h3 id="indices"><a class="anchor" href="#indices">¶</a>Indices</h3>
258<p>In order to execute our geospatial queries we have to create an index on the attribute that represents the location, because the operators that we will use requires it. This attribute can be a <a href="https://docs.mongodb.com/manual/reference/geojson/">GeoJSON object</a> or a legacy coordinate pair.</p>
259<p>We have decided to use a GeoJSON object because we want to avoid legacy features that may be deprecated in the future.</p>
260<p>The attribute is called <code>geo</code> for the <code>Tree</code> and <code>Noise</code> objects and <code>start</code>, <code>middle</code> or <code>end</code> for the <code>Via</code> class. In the <code>Via</code> we are going to index the attribute <code>middle</code> because it is the most representative field for us. Because the <code>Via</code> is inside the <code>Census</code> and it doesn’t have its own collection, we create the index on the <code>Census</code> collection.</p>
261<p>The used index type is <code>2dsphere</code> because it supports queries that work on geometries on an earth-like sphere. Another option is the <code>2d</code> index but it’s not a good fit for our because it is for queries that calculate geometries on a two-dimensional plane.</p>
262<h3 id="running_the_server"><a class="anchor" href="#running_the_server">¶</a>Running the server</h3>
263<p>If we ignore the configuration part of the server creation, our <code>server.py</code> file is pretty simple. Its job is to create a <a href="https://aiohttp.readthedocs.io/en/stable/web.html">server application</a>, setup Mongo and return it to the caller so that they can run it:</p>
264<pre><code>import asyncio
265import subprocess
266import motor.motor_asyncio
267
268from aiohttp import web
269
270from . import rest, data
271
272def create_app():
273 ret = subprocess.run('npm run build', cwd='../client', shell=True).returncode
274 if ret != 0:
275 exit(ret)
276
277 db = motor.motor_asyncio.AsyncIOMotorClient().opendata
278 loop = asyncio.get_event_loop()
279 loop.run_until_complete(data.insert_to_db(db))
280
281 app = web.Application()
282 app['db'] = db
283
284 app.router.add_routes([
285 web.get('/', lambda r: web.HTTPSeeOther('/index.html')),
286 *rest.ROUTES,
287 web.static('/', os.path.join(config['www']['root'], 'public')),
288 ])
289
290 return app
291</code></pre>
292<p>There’s a bit going on here, but it’s nothing too complex:</p>
293<ul>
294<li>We automatically run <code>npm run build</code> on the frontend because it’s very comfortable to have the frontend built automatically before the server runs.</li>
295<li>We create a Motor client and access the <code>opendata</code> database. Into it, we load the data, effectively saving it in Mongo for the server to use.</li>
296<li>We create the server application and save a reference to the Mongo database in it, so that it can be used later on any endpoint without needing to recreate it.</li>
297<li>We define the routes of our app: root, REST and static (where the frontend files live). We’ll get to the <code>rest</code> part soon.
298Running the server is now simple:</li>
299</ul>
300<pre><code>def main():
301 from aiohttp import web
302 from . import server
303
304 app = server.create_app()
305 web.run_app(app)
306
307if __name__ == '__main__':
308 main()
309</code></pre>
310<h3 id="rest_endpoints"><a class="anchor" href="#rest_endpoints">¶</a>REST endpoints</h3>
311<p>The frontend will communicate with the backend via <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">REST</a> calls, so that it can ask for things like «give me the information associated with this area», and the web server can query the Mongo server to reply with a HTTP response. This little diagram should help:</p>
312<p><img src="bitmap.png" alt="" /></p>
313<p>What we need to do, then, is define those REST endpoints we mentioned earlier when creating the server. We will process the HTTP request, ask Mongo for the data, and return the HTTP response:</p>
314<pre><code>import asyncio
315import pymongo
316
317from aiohttp import web
318
319async def get_area_info(request):
320 try:
321 long = float(request.query['long'])
322 lat = float(request.query['lat'])
323 distance = float(request.query['distance'])
324 except KeyError as e:
325 raise web.HTTPBadRequest(reason=f'a required parameter was missing: {e.args[0]}')
326 except ValueError:
327 raise web.HTTPBadRequest(reason='one of the parameters was not a valid float')
328
329 geo_avg_noise_pipeline = [{
330 '$geoNear': {
331 'near' : {'type': 'Point', 'coordinates': [long, lat]},
332 'maxDistance': distance,
333 'minDistance': 0,
334 'spherical' : 'true',
335 'distanceField' : 'distance'
336 }
337 }]
338
339 db = request.app['db']
340
341 try:
342 noise_count, sum_noise, avg_noise = 0, 0, 0
343 async for item in db.noise.aggregate(geo_avg_noise_pipeline):
344 noise_count += 1
345 sum_noise += item['level']
346
347 if noise_count != 0:
348 avg_noise = sum_noise / noise_count
349 else:
350 avg_noise = None
351
352 except pymongo.errors.ConnectionFailure:
353 raise web.HTTPServiceUnavailable(reason='no connection to database')
354
355 return web.json_response({
356 'tree_count': tree_count,
357 'trees_per_type': [[k, v] for k, v in trees_per_type.items()],
358 'census_count': census_count,
359 'avg_noise': avg_noise,
360 })
361
362ROUTES = [
363 web.get('/rest/get-area-info', get_area_info)
364]
365</code></pre>
366<p>In this code, we’re only showing how to return the average noise because that’s the simplest we can do. The real code also fetches tree count, tree count per type, and census count.</p>
367<p>Again, there’s quite a bit to go through, so let’s go step by step:</p>
368<ul>
369<li>We parse the frontend’s <code>request.query</code> into <code>float</code> that we can use. In particular, the frontend is asking us for information at a certain latitude, longitude, and distance. If the query is malformed, we return a proper error.</li>
370<li>We create our query for Mongo outside, just so it’s clearer to read.</li>
371<li>We access the database reference we stored earlier when creating the server with <code>request.app['db']</code>. Handy!</li>
372<li>We try to query Mongo. It may fail if the Mongo server is not running, so we should handle that and tell the client what’s happening. If it succeeds though, we will gather information about the average noise.</li>
373<li>We return a <code>json_response</code> with Mongo results for the frontend to present to the user.
374You may have noticed we defined a <code>ROUTES</code> list at the bottom. This will make it easier to expand in the future, and the server creation won’t need to change anything in its code, because it’s already unpacking all the routes we define here.</li>
375</ul>
376<h3 id="geospatial_queries"><a class="anchor" href="#geospatial_queries">¶</a>Geospatial queries</h3>
377<p>In order to retrieve the information from Mongo database we have defined two geospatial queries:</p>
378<pre><code>geo_query = {
379 '$nearSphere' : {
380 '$geometry': {
381 'type': 'Point',
382 'coordinates': [long, lat]
383 },
384 '$maxDistance': distance,
385 '$minDistance': 0
386 }
387}
388</code></pre>
389<p>This query uses <a href="https://docs.mongodb.com/manual/reference/operator/query/nearSphere/#op._S_nearSphere">the operator <code>$nearSphere</code></a> which return geospatial objects in proximity to a point on a sphere.</p>
390<p>The sphere point is represented by the <code>$geometry</code> operator where it is specified the type of geometry and the coordinates (given by the HTTP request).</p>
391<p>The maximum and minimum distance are represented by <code>$maxDistance</code> and <code>$minDistance</code> respectively. We specify that the maximum distance is the radio selected by the user.</p>
392<pre><code>geo_avg_noise_pipeline = [{
393 '$geoNear': {
394 'near' : {'type': 'Point', 'coordinates': [long, lat]},
395 'maxDistance': distance,
396 'minDistance': 0,
397 'spherical' : 'true',
398 'distanceField' : 'distance'
399 }
400}]
401</code></pre>
402<p>This query uses the <a href="https://docs.mongodb.com/manual/core/aggregation-pipeline/">aggregation pipeline</a> stage <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/geoNear/#pipe._S_geoNear"><code>$geoNear</code></a> which returns an ordered stream of documents based on the proximity to a geospatial point. The output documents include an additional distance field.</p>
403<p>The <code>near</code> field is mandatory and is the point for which to find the closest documents. In this field it is specified the type of geometry and the coordinates (given by the HTTP request).</p>
404<p>The <code>distanceField</code> field is also mandatory and is the output field that will contain the calculated distance. In this case we’ve just called it <code>distance</code>.</p>
405<p>Some other fields are <code>maxDistance</code> that indicates the maximum allowed distance from the center of the point, <code>minDistance</code> for the minimum distance, and <code>spherical</code> which tells MongoDB how to calculate the distance between two points.</p>
406<p>We specify the maximum distance as the radio selected by the user in the frontend.</p>
407<h2 id="frontend"><a class="anchor" href="#frontend">¶</a>Frontend</h2>
408<p>As said earlier, our frontend will use Svelte. We already downloaded the template, so we can start developing. For some, this is the most fun part, because they can finally see and interact with some of the results. But for this interaction to work, we needed a functional backend which we now have!</p>
409<h3 id="rest_queries"><a class="anchor" href="#rest_queries">¶</a>REST queries</h3>
410<p>The frontend has to query the server to get any meaningful data to show on the page. The <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">Fetch API</a> does not throw an exception if the server doesn’t respond with HTTP OK, but we would like one if things go wrong, so that we can handle them gracefully. The first we’ll do is define our own exception <a href="https://stackoverflow.com/a/27724419">which is not pretty</a>:</p>
411<pre><code>function NetworkError(message, status) {
412 var instance = new Error(message);
413 instance.name = 'NetworkError';
414 instance.status = status;
415 Object.setPrototypeOf(instance, Object.getPrototypeOf(this));
416 if (Error.captureStackTrace) {
417 Error.captureStackTrace(instance, NetworkError);
418 }
419 return instance;
420}
421
422NetworkError.prototype = Object.create(Error.prototype, {
423 constructor: {
424 value: Error,
425 enumerable: false,
426 writable: true,
427 configurable: true
428 }
429});
430Object.setPrototypeOf(NetworkError, Error);
431</code></pre>
432<p>But hey, now we have a proper and reusable <code>NetworkError</code>! Next, let’s make a proper and reusabe <code>query</code> function that deals with <code>fetch</code> for us:</p>
433<pre><code>async function query(endpoint) {
434 const res = await fetch(endpoint, {
435 // if we ever use cookies, this is important
436 credentials: 'include'
437 });
438 if (res.ok) {
439 return await res.json();
440 } else {
441 throw new NetworkError(await res.text(), res.status);
442 }
443}
444</code></pre>
445<p>At last, we can query our web server. The export here tells Svelte that this function should be visible to outer modules (public) as opposed to being private:</p>
446<pre><code>export function get_area_info(long, lat, distance) {
447 return query(`/rest/get-area-info?long=${long}&lat=${lat}&distance=${distance}`);
448}
449</code></pre>
450<p>The attentive reader will have noticed that <code>query</code> is <code>async</code>, but <code>get_area_info</code> is not. This is intentional, because we don’t need to <code>await</code> for anything inside of it. We can just return the <code>[Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)</code> that <code>query</code> created and let the caller <code>await</code> it as they see fit. The <code>await</code> here would have been redundant.</p>
451<p>For those of you who don’t know what a JavaScript promise is, think of it as an object that represents «an eventual result». The result may not be there yet, but we promised it will be present in the future, and we can <code>await</code> for it. You can also find the same concept in other languages like Python under a different name, such as <a href="https://docs.python.org/3/library/asyncio-future.html#asyncio.Future"><code>Future</code></a>.</p>
452<h3 id="map_component"><a class="anchor" href="#map_component">¶</a>Map component</h3>
453<p>In Svelte, we can define self-contained components that are issolated from the rest. This makes it really easy to create a modular application. Think of a Svelte component as your own HTML tag, which you can customize however you want, building upon the already-existing components HTML has to offer.</p>
454<p>The main thing that our map needs to do is render the map as an image and overlay the selection area as the user hovers the map with their mouse. We could render the image in the canvas itself, but instead we’ll use the HTML <code><img></code> tag for that and put a transparent <code><canvas></code> on top with some CSS. This should make it cheaper and easier to render things on the canvas.</p>
455<p>The <code>Map</code> component will thus render as the user moves the mouse over it, and produce an event when they click so that whatever component is using a <code>Map</code> knows that it was clicked. Here’s the final CSS and HTML:</p>
456<pre><code><style>
457div {
458 position: relative;
459}
460canvas {
461 position: absolute;
462 left: 0;
463 top: 0;
464 cursor: crosshair;
465}
466</style>
467
468<div>
469 <img bind:this={img} on:load={handleLoad} {height} src="caceres-municipality.svg" alt="Cáceres (municipality)"/>
470 <canvas
471 bind:this={canvas}
472 on:mousemove={handleMove}
473 on:wheel={handleWheel}
474 on:mouseup={handleClick}/>
475</div>
476</code></pre>
477<p>We hardcode a map source here, but ideally this would be provided by the server. The project is already complex enough, so we tried to avoid more complexity than necessary.</p>
478<p>We bind the tags to some variables declared in the JavaScript code of the component, along with some functions and parameters to let the users of <code>Map</code> customize it just a little.</p>
479<p>Here’s the gist of the JavaScript code:</p>
480<pre><code><script>
481 import { createEventDispatcher, onMount } from 'svelte';
482
483 export let height = 200;
484
485 const dispatch = createEventDispatcher();
486
487 let img;
488 let canvas;
489
490 const LONG_WEST = -6.426881;
491 const LONG_EAST = -6.354143;
492 const LAT_NORTH = 39.500064;
493 const LAT_SOUTH = 39.443201;
494
495 let x = 0;
496 let y = 0;
497 let clickInfo = null; // [x, y, radius]
498 let radiusDelta = 0.005 * height;
499 let maxRadius = 0.2 * height;
500 let minRadius = 0.01 * height;
501 let radius = 0.05 * height;
502
503 function handleLoad() {
504 canvas.width = img.width;
505 canvas.height = img.height;
506 }
507
508 function handleMove(event) {
509 const { left, top } = this.getBoundingClientRect();
510 x = Math.round(event.clientX - left);
511 y = Math.round(event.clientY - top);
512 }
513
514 function handleWheel(event) {
515 if (event.deltaY < 0) {
516 if (radius < maxRadius) {
517 radius += radiusDelta;
518 }
519 } else {
520 if (radius > minRadius) {
521 radius -= radiusDelta;
522 }
523 }
524 event.preventDefault();
525 }
526
527 function handleClick(event) {
528 dispatch('click', {
529 // the real code here maps the x/y/radius values to the right range, here omitted
530 x: ...,
531 y: ...,
532 radius: ...,
533 });
534 }
535
536 onMount(() => {
537 const ctx = canvas.getContext('2d');
538 let frame;
539
540 (function loop() {
541 frame = requestAnimationFrame(loop);
542
543 // the real code renders mouse area/selection, here omitted for brevity
544 ...
545 }());
546
547 return () => {
548 cancelAnimationFrame(frame);
549 };
550 });
551</script>
552</code></pre>
553<p>Let’s go through bit-by-bit:</p>
554<ul>
555<li>We define a few variables and constants for later use in the final code.</li>
556<li>We define the handlers to react to mouse movement and clicks. On click, we dispatch an event to outer components.</li>
557<li>We setup the render loop with animation frames, and cancel the current frame appropriatedly if the component disappears.</li>
558</ul>
559<h3 id="app_component"><a class="anchor" href="#app_component">¶</a>App component</h3>
560<p>Time to put everything together! We wil include our function to make REST queries along with our <code>Map</code> component to render things on screen.</p>
561<pre><code><script>
562 import Map from './Map.svelte';
563 import { get_area_info } from './rest.js'
564 let selection = null;
565 let area_info_promise = null;
566 function handleMapSelection(event) {
567 selection = event.detail;
568 area_info_promise = get_area_info(selection.x, selection.y, selection.radius);
569 }
570 function format_avg_noise(avg_noise) {
571 if (avg_noise === null) {
572 return '(no data)';
573 } else {
574 return `${avg_noise.toFixed(2)} dB`;
575 }
576 }
577</script>
578
579<div class="container-fluid">
580 <div class="row">
581 <div class="col-3" style="max-width: 300em;">
582 <div class="text-center">
583 <h1>Caceres Data Consultory</h1>
584 </div>
585 <Map height={400} on:click={handleMapSelection}/>
586 <div class="text-center mt-4">
587 {#if selection === null}
588 <p class="m-1 p-3 border border-bottom-0 bg-info text-white">Click on the map to select the area you wish to see details for.</p>
589 {:else}
590 <h2 class="bg-dark text-white">Selected area</h2>
591 <p><b>Coordinates:</b> ({selection.x}, {selection.y})</p>
592 <p><b>Radius:</b> {selection.radius} meters</p>
593 {/if}
594 </div>
595 </div>
596 <div class="col-sm-4">
597 <div class="row">
598 {#if area_info_promise !== null}
599 {#await area_info_promise}
600 <p>Fetching area information…</p>
601 {:then area_info}
602 <div class="col">
603 <div class="text-center">
604 <h2 class="m-1 bg-dark text-white">Area information</h2>
605 <ul class="list-unstyled">
606 <li>There are <b>{area_info.tree_count} trees </b> within the area</li>
607 <li>The <b>average noise</b> is <b>{format_avg_noise(area_info.avg_noise)}</b></li>
608 <li>There are <b>{area_info.census_count} persons </b> within the area</li>
609 </ul>
610 </div>
611 {#if area_info.trees_per_type.length > 0}
612 <div class="text-center">
613 <h2 class="m-1 bg-dark text-white">Tree count per type</h2>
614 </div>
615 <ul class="list-group">
616 {#each area_info.trees_per_type as [type, count]}
617 <li class="list-group-item">{type} <span class="badge badge-dark float-right">{count}</span></li>
618 {/each}
619 </ul>
620 {/if}
621 </div>
622 {:catch error}
623 <p>Failed to fetch area information: {error.message}</p>
624 {/await}
625 {/if}
626 </div>
627 </div>
628 </div>
629</div>
630</code></pre>
631<ul>
632<li>We import the <code>Map</code> component and REST function so we can use them.</li>
633<li>We define a listener for the events that the <code>Map</code> produces. Such event will trigger a REST call to the server and save the result in a promise used later.</li>
634<li>We’re using Bootstrap for the layout because it’s a lot easier. In the body we add our <code>Map</code> and another column to show the selection information.</li>
635<li>We make use of Svelte’s <code>{#await}</code> to nicely notify the user when the call is being made, when it was successful, and when it failed. If it’s successful, we display the info.</li>
636</ul>
637<h2 id="results"><a class="anchor" href="#results">¶</a>Results</h2>
638<p>Lo and behold, watch our application run!</p>
639<p><video controls="controls" src="sr-2020-04-14_09-28-25.mp4"></video></p>
640<p>In this video you can see our application running, but let’s describe what is happening in more detail.</p>
641<p>When the application starts running (by opening it in your web browser of choice), you can see a map with the town of Cáceres. Then you, the user, can click to retrieve the information within the selected area.</p>
642<p>It is important to note that one can make the selection area larger or smaller by trying to scroll up or down, respectively.</p>
643<p>Once an area is selected, it is colored green in order to let the user know which area they have selected. Under the map, the selected coordinates and the radius (in meters) is also shown for the curious. At the right side the information concerning the selected area is shown, such as the number of trees, the average noise and the number of persons. If there are trees in the area, the application also displays the trees per type, sorted by the number of trees.</p>
644<h2 id="download"><a class="anchor" href="#download">¶</a>Download</h2>
645<p>We hope you enjoyed reading this post as much as we enjoyed writing it! Feel free to download the final project and play around with it. Maybe you can adapt it for even more interesting purposes!</p>
646<p><em>download removed</em></p>
647<p>To run the above code:</p>
648<ol>
649<li>Unzip the downloaded file.</li>
650<li>Make a copy of <code>example-server-config.ini</code> and rename it to <code>server-config.ini</code>, then edit the file to suit your needs.</li>
651<li>Run the server with <code>python -m server</code>.</li>
652<li>Open <a href="http://localhost:9000">localhost:9000</a> in your web browser (or whatever port you chose) and enjoy!</li>
653</ol>
654</main>
655</body>
656</html>
657