blog/ribw/developing-a-python-application-for-mongodb/index.html (view raw)
1<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=description content="Official Lonami's website"><meta name=viewport content="width=device-width, initial-scale=1.0, user-scalable=yes"><title> Developing a Python application for MongoDB | Lonami's Blog </title><link rel=stylesheet href=/style.css><body><article><nav class=sections><ul class=left><li><a href=/>lonami's site</a><li><a href=/blog class=selected>blog</a><li><a href=/golb>golb</a></ul><div class=right><a href=https://github.com/LonamiWebs><img src=/img/github.svg alt=github></a><a href=/blog/atom.xml><img src=/img/rss.svg alt=rss></a></div></nav><main><h1 class=title>Developing a Python application for MongoDB</h1><div class=time><p>2020-03-25T00:00:04+00:00<p>last updated 2020-04-16T08:01:23+00:00</div><p>This is the third and last post in the MongoDB series, where we will develop a Python application to process and store OpenData inside Mongo.<p>Other posts in this series:<ul><li><a href=/blog/ribw/mongodb-an-introduction/>MongoDB: an Introduction</a><li><a href=/blog/ribw/mongodb-basic-operations-and-architecture/>MongoDB: Basic Operations and Architecture</a><li><a href=/blog/ribw/developing-a-python-application-for-mongodb/>Developing a Python application for MongoDB</a> (this post)</ul><p>This post is co-authored wih a Classmate.<hr><h2 id=what-are-we-making>What are we making?</h2><p>We are going to develop a web application that renders a map, in this case, the town of Cáceres, with which users can interact. When the user clicks somewhere on the map, the selected location will be sent to the server to process. This server will perform geospatial queries to Mongo and once the results are ready, the information is presented back at the webpage.<p>The data used for the application comes from <a href=https://opendata.caceres.es/>Cáceres’ OpenData</a>, and our goal is that users will be able to find information about certain areas in a quick and intuitive way, such as precise coordinates, noise level, and such.<h2 id=what-are-we-using>What are we using?</h2><p>The web application will be using <a href=https://python.org/>Python</a> for the backend, <a href=https://svelte.dev/>Svelte</a> for the frontend, and <a href=https://www.mongodb.com/>Mongo</a> as our storage database and processing center.<ul><li><strong>Why Python?</strong> It’s a comfortable language to write and to read, and has a great ecosystem with <a href=https://pypi.org/>plenty of libraries</a>.<li><strong>Why Svelte?</strong> Svelte is the New Thing<strong>™</strong> in the world of component frameworks for JavaScript. It is similar to React or Vue, but compiled and with a lot less boilerplate. Check out their <a href=https://svelte.dev/blog/svelte-3-rethinking-reactivity>Svelte post</a> to learn more.<li><strong>Why Mongo?</strong> We believe NoSQL is the right approach for doing the kind of processing and storage that we expect, and it’s <a href=https://docs.mongodb.com/>very easy to use</a>. In addition, we will be making Geospatial Queries which <a href=https://docs.mongodb.com/manual/geospatial-queries/>Mongo supports</a>.</ul><p>Why didn’t we choose to make a smaller project, you may ask? You will be shocked to hear that we do not have an answer for that!<p>Note that we will not be embedding <strong>all</strong> the code of the project in this post, or it would be too long! We will include only the relevant snippets needed to understand the core ideas of the project, and not the unnecessary parts of it (for example, parsing configuration files to easily change the port where the server runs is not included).<h2 id=python-dependencies>Python dependencies</h2><p>Because we will program it in Python, you need Python installed. You can install it using a package manager of your choice or heading over to the <a href=https://www.python.org/downloads/>Python downloads section</a>, but if you’re on Linux, chances are you have it installed already.<p>Once Python 3.7 or above is installed, install <a href=https://motor.readthedocs.io/en/stable/><code>motor</code> (Asynchronous Python driver for MongoDB)</a> and the <a href=https://docs.aiohttp.org/en/stable/web.html><code>aiohttp</code> server</a> through <code>pip</code>:<pre><code>pip install aiohttp motor
2</code></pre><p>Make sure that Mongo is running in the background (this has been described in previous posts), and we should be able to get to work.<h2 id=web-dependencies>Web dependencies</h2><p>To work with Svelte and its dependencies, we will need <code>[npm](https://www.npmjs.com/)</code> which comes with <a href=https://nodejs.org/en/>NodeJS</a>, so go and <a href=https://nodejs.org/en/download/>install Node from their site</a>. The download will be different depending on your operating system.<p>Following <a href=https://svelte.dev/blog/the-easiest-way-to-get-started>the easiest way to get started with Svelte</a>, we will put our project in a <code>client/</code> folder (because this is what the clients see, the frontend). Feel free to tinker a bit with the configuration files to change the name and such, although this isn’t relevant for the rest of the post.<h2 id=finding-the-data>Finding the data</h2><p>We are going to work with the JSON files provided by <a href=http://opendata.caceres.es/>OpenData Cáceres</a>. In particular, we want information about the noise, census, vias and trees. To save you the time from <a href=http://opendata.caceres.es/dataset>searching each of these</a>, we will automate the download with code.<p>If you want to save the data offline or just know what data we’ll be using for other purposes though, you can right click on the following links and select «Save Link As…» with the name of the link:<ul><li><code>[noise.json](http://opendata.caceres.es/GetData/GetData?dataset=om:MedicionRuido&format=json)</code><li><code>[census.json](http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionPadron&year=2017&format=json)</code><li><code>[vias.json](http://opendata.caceres.es/GetData/GetData?dataset=om:InformacionPadron&year=2017&format=json)</code><li><code>[trees.json](http://opendata.caceres.es/GetData/GetData?dataset=om:Arbol&format=json)</code></ul><h2 id=backend>Backend</h2><p>It’s time to get started with some code! We will put it in a <code>server/</code> folder because it will contain the Python server, that is, the backend of our application.<p>We are using <code>aiohttp</code> because we would like our server to be <code>async</code>. We don’t expect a lot of users at the same time, but it’s good to know our server would be well-designed for that use-case. As a bonus, it makes IO points clear in the code, which can help reason about it. The implicit synchronization between <code>await</code> is also a nice bonus.<h3 id=saving-the-data-in-mongo>Saving the data in Mongo</h3><p>Before running the server, we must ensure that the data we need is already stored and indexed in Mongo. Our <code>server/data.py</code> will take care of downloading the files, cleaning them up a little (Cáceres’ OpenData can be a bit awkward sometimes), inserting them into Mongo and indexing them.<p>Downloading the JSON data can be done with <code>[ClientSession.get](https://aiohttp.readthedocs.io/en/stable/client_reference.html#aiohttp.ClientSession.get)</code>. We also take this opportunity to clean up the messy encoding from the JSON, which does not seem to be UTF-8 in some cases.<pre><code>async def load_json(session, url):
3 fixes = [(old, new.encode('utf-8')) for old, new in [
4 (b'\xc3\x83\\u2018', 'Ñ'),
5 (b'\xc3\x83\\u0081', 'Á'),
6 (b'\xc3\x83\\u2030', 'É'),
7 (b'\xc3\x83\\u008D', 'Í'),
8 (b'\xc3\x83\\u201C', 'Ó'),
9 (b'\xc3\x83\xc5\xa1', 'Ú'),
10 (b'\xc3\x83\xc2\xa1', 'á'),
11 ]]
12
13 async with session.get(url) as resp:
14 data = await resp.read()
15
16 # Yes, this feels inefficient, but it's not really worth improving.
17 for old, new in fixes:
18 data = data.replace(old, new)
19
20 data = data.decode('utf-8')
21 return json.loads(data)
22</code></pre><p>Later on, it can be reused for the various different URLs:<pre><code>import aiohttp
23
24NOISE_URL = 'http://opendata.caceres.es/GetData/GetData?dataset=om:MedicionRuido&format=json'
25# (...other needed URLs here)
26
27async def insert_to_db(db):
28 async with aiohttp.ClientSession() as session:
29 data = await load_json(session, NOISE_URL)
30 # now we have the JSON data cleaned up, ready to be parsed
31</code></pre><h3 id=data-model>Data model</h3><p>With the JSON data in our hands, it’s time to parse it. Always remember to <a href=https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/>parse, don’t validate</a>. With <a href=https://docs.python.org/3/library/dataclasses.html>Python 3.7 <code>dataclasses</code></a> it’s trivial to define classes that will store only the fields we care about, typed, and with proper names:<pre><code>from dataclasses import dataclass
32
33Longitude = float
34Latitude = float
35
36@dataclass
37class GSON:
38 type: str
39 coordinates: (Longitude, Latitude)
40
41@dataclass
42class Noise:
43 id: int
44 geo: GSON
45 level: float
46</code></pre><p>This makes it really easy to see that, if we have a <code>Noise</code>, we can access its <code>geo</code> data which is a <code>GSON</code> with a <code>type</code> and <code>coordinates</code>, having <code>Longitude</code> and <code>Latitude</code> respectively. <code>dataclasses</code> and <code>[typing](https://docs.python.org/3/library/typing.html)</code> make dealing with this very easy and clear.<p>Every dataclass will be on its own collection inside Mongo, and these are:<ul><li><p>Noise<li><p>Integer <code>id</code><li><p>GeoJSON <code>geo</code><li><p>String <code>type</code><li><p>Longitude-latitude pair <code>coordinates</code><li><p>Floating-point number <code>level</code><li><p>Tree<li><p>String <code>name</code><li><p>String <code>gender</code><li><p>Integer <code>units</code><li><p>Floating-point number <code>height</code><li><p>Floating-point number <code>cup_diameter</code><li><p>Floating-point number <code>trunk_diameter</code><li><p>Optional string <code>variety</code><li><p>Optional string <code>distribution</code><li><p>GeoJSON <code>geo</code><li><p>Optional string <code>irrigation</code><li><p>Census<li><p>Integer <code>year</code><li><p>Via <code>via</code><li><p>String <code>name</code><li><p>String <code>kind</code><li><p>Integer <code>code</code><li><p>Optional string <code>history</code><li><p>Optional string <code>old_name</code><li><p>Optional floating-point number <code>length</code><li><p>Optional GeoJSON <code>start</code><li><p>GeoJSON <code>middle</code><li><p>Optional GeoJSON <code>end</code><li><p>Optional list with geometry pairs <code>geometry</code><li><p>Integer <code>count</code><li><p>Mapping year-to-count <code>count_per_year</code><li><p>Mapping gender-to-count <code>count_per_gender</code><li><p>Mapping nationality-to-count <code>count_per_nationality</code><li><p>Integer <code>time_year</code></ul><p>Now, let’s define a method to actually parse the JSON and yield instances from these new data classes:<pre><code>@classmethod
47def iter_from_json(cls, data):
48 for row in data['results']['bindings']:
49 noise_id = int(row['uri']['value'].split('/')[-1])
50 long = float(row['geo_long']['value'])
51 lat = float(row['geo_lat']['value'])
52 level = float(row['om_nivelRuido']['value'])
53
54 yield cls(
55 id=noise_id,
56 geo=GSON(type='Point', coordinates=[long, lat]),
57 level=level
58 )
59</code></pre><p>Here we iterate over the input JSON <code>data</code> bindings and <code>yield cls</code> instances with more consistent naming than the original one. We also extract the data from the many unnecessary nested levels of the JSON and have something a lot flatter to work with.<p>For those of you who don’t know what <code>yield</code> does (after all, not everyone is used to seeing generators), here’s two functions that work nearly the same:<pre><code>def squares_return(n):
60 result = []
61 for i in range(n):
62 result.append(n ** 2)
63 return result
64
65def squares_yield(n):
66 for i in range(n):
67 yield n ** 2
68</code></pre><p>The difference is that the one with <code>yield</code> is «lazy» and doesn’t need to do all the work up-front. It will generate (yield) more values as they are needed when you use a <code>for</code> loop. Generally, it’s a better idea to create generator functions than do all the work early which may be unnecessary. See <a href=https://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do>What does the «yield» keyword do?</a> if you still have questions.<p>With everything parsed, it’s time to insert the data into Mongo. If the data was not present yet (0 documents), then we will download the file, parse it, insert it as documents into the given Mongo <code>db</code>, and index it:<pre><code>from dataclasses import asdict
69
70async def insert_to_db(db):
71 async with aiohttp.ClientSession() as session:
72 if await db.noise.estimated_document_count() == 0:
73 data = await load_json(session, NOISE_URL)
74
75 await db.noise.insert_many(asdict(noise) for noise in Noise.iter_from_json(data))
76 await db.noise.create_index([('geo', '2dsphere')])
77</code></pre><p>We repeat this process for all the other data, and just like that, Mongo is ready to be used in our server.<h3 id=indices>Indices</h3><p>In order to execute our geospatial queries we have to create an index on the attribute that represents the location, because the operators that we will use requires it. This attribute can be a <a href=https://docs.mongodb.com/manual/reference/geojson/>GeoJSON object</a> or a legacy coordinate pair.<p>We have decided to use a GeoJSON object because we want to avoid legacy features that may be deprecated in the future.<p>The attribute is called <code>geo</code> for the <code>Tree</code> and <code>Noise</code> objects and <code>start</code>, <code>middle</code> or <code>end</code> for the <code>Via</code> class. In the <code>Via</code> we are going to index the attribute <code>middle</code> because it is the most representative field for us. Because the <code>Via</code> is inside the <code>Census</code> and it doesn’t have its own collection, we create the index on the <code>Census</code> collection.<p>The used index type is <code>2dsphere</code> because it supports queries that work on geometries on an earth-like sphere. Another option is the <code>2d</code> index but it’s not a good fit for our because it is for queries that calculate geometries on a two-dimensional plane.<h3 id=running-the-server>Running the server</h3><p>If we ignore the configuration part of the server creation, our <code>server.py</code> file is pretty simple. Its job is to create a <a href=https://aiohttp.readthedocs.io/en/stable/web.html>server application</a>, setup Mongo and return it to the caller so that they can run it:<pre><code>import asyncio
78import subprocess
79import motor.motor_asyncio
80
81from aiohttp import web
82
83from . import rest, data
84
85def create_app():
86 ret = subprocess.run('npm run build', cwd='../client', shell=True).returncode
87 if ret != 0:
88 exit(ret)
89
90 db = motor.motor_asyncio.AsyncIOMotorClient().opendata
91 loop = asyncio.get_event_loop()
92 loop.run_until_complete(data.insert_to_db(db))
93
94 app = web.Application()
95 app['db'] = db
96
97 app.router.add_routes([
98 web.get('/', lambda r: web.HTTPSeeOther('/index.html')),
99 *rest.ROUTES,
100 web.static('/', os.path.join(config['www']['root'], 'public')),
101 ])
102
103 return app
104</code></pre><p>There’s a bit going on here, but it’s nothing too complex:<ul><li>We automatically run <code>npm run build</code> on the frontend because it’s very comfortable to have the frontend built automatically before the server runs.<li>We create a Motor client and access the <code>opendata</code> database. Into it, we load the data, effectively saving it in Mongo for the server to use.<li>We create the server application and save a reference to the Mongo database in it, so that it can be used later on any endpoint without needing to recreate it.<li>We define the routes of our app: root, REST and static (where the frontend files live). We’ll get to the <code>rest</code> part soon. Running the server is now simple:</ul><pre><code>def main():
105 from aiohttp import web
106 from . import server
107
108 app = server.create_app()
109 web.run_app(app)
110
111if __name__ == '__main__':
112 main()
113</code></pre><h3 id=rest-endpoints>REST endpoints</h3><p>The frontend will communicate with the backend via <a href=https://en.wikipedia.org/wiki/Representational_state_transfer>REST</a> calls, so that it can ask for things like «give me the information associated with this area», and the web server can query the Mongo server to reply with a HTTP response. This little diagram should help:<p><img src=https://lonami.dev/blog/ribw/developing-a-python-application-for-mongodb/bitmap.png><p>What we need to do, then, is define those REST endpoints we mentioned earlier when creating the server. We will process the HTTP request, ask Mongo for the data, and return the HTTP response:<pre><code>import asyncio
114import pymongo
115
116from aiohttp import web
117
118async def get_area_info(request):
119 try:
120 long = float(request.query['long'])
121 lat = float(request.query['lat'])
122 distance = float(request.query['distance'])
123 except KeyError as e:
124 raise web.HTTPBadRequest(reason=f'a required parameter was missing: {e.args[0]}')
125 except ValueError:
126 raise web.HTTPBadRequest(reason='one of the parameters was not a valid float')
127
128 geo_avg_noise_pipeline = [{
129 '$geoNear': {
130 'near' : {'type': 'Point', 'coordinates': [long, lat]},
131 'maxDistance': distance,
132 'minDistance': 0,
133 'spherical' : 'true',
134 'distanceField' : 'distance'
135 }
136 }]
137
138 db = request.app['db']
139
140 try:
141 noise_count, sum_noise, avg_noise = 0, 0, 0
142 async for item in db.noise.aggregate(geo_avg_noise_pipeline):
143 noise_count += 1
144 sum_noise += item['level']
145
146 if noise_count != 0:
147 avg_noise = sum_noise / noise_count
148 else:
149 avg_noise = None
150
151 except pymongo.errors.ConnectionFailure:
152 raise web.HTTPServiceUnavailable(reason='no connection to database')
153
154 return web.json_response({
155 'tree_count': tree_count,
156 'trees_per_type': [[k, v] for k, v in trees_per_type.items()],
157 'census_count': census_count,
158 'avg_noise': avg_noise,
159 })
160
161ROUTES = [
162 web.get('/rest/get-area-info', get_area_info)
163]
164</code></pre><p>In this code, we’re only showing how to return the average noise because that’s the simplest we can do. The real code also fetches tree count, tree count per type, and census count.<p>Again, there’s quite a bit to go through, so let’s go step by step:<ul><li>We parse the frontend’s <code>request.query</code> into <code>float</code> that we can use. In particular, the frontend is asking us for information at a certain latitude, longitude, and distance. If the query is malformed, we return a proper error.<li>We create our query for Mongo outside, just so it’s clearer to read.<li>We access the database reference we stored earlier when creating the server with <code>request.app['db']</code>. Handy!<li>We try to query Mongo. It may fail if the Mongo server is not running, so we should handle that and tell the client what’s happening. If it succeeds though, we will gather information about the average noise.<li>We return a <code>json_response</code> with Mongo results for the frontend to present to the user. You may have noticed we defined a <code>ROUTES</code> list at the bottom. This will make it easier to expand in the future, and the server creation won’t need to change anything in its code, because it’s already unpacking all the routes we define here.</ul><h3 id=geospatial-queries>Geospatial queries</h3><p>In order to retrieve the information from Mongo database we have defined two geospatial queries:<pre><code>geo_query = {
165 '$nearSphere' : {
166 '$geometry': {
167 'type': 'Point',
168 'coordinates': [long, lat]
169 },
170 '$maxDistance': distance,
171 '$minDistance': 0
172 }
173}
174</code></pre><p>This query uses <a href=https://docs.mongodb.com/manual/reference/operator/query/nearSphere/#op._S_nearSphere>the operator <code>$nearSphere</code></a> which return geospatial objects in proximity to a point on a sphere.<p>The sphere point is represented by the <code>$geometry</code> operator where it is specified the type of geometry and the coordinates (given by the HTTP request).<p>The maximum and minimum distance are represented by <code>$maxDistance</code> and <code>$minDistance</code> respectively. We specify that the maximum distance is the radio selected by the user.<pre><code>geo_avg_noise_pipeline = [{
175 '$geoNear': {
176 'near' : {'type': 'Point', 'coordinates': [long, lat]},
177 'maxDistance': distance,
178 'minDistance': 0,
179 'spherical' : 'true',
180 'distanceField' : 'distance'
181 }
182}]
183</code></pre><p>This query uses the <a href=https://docs.mongodb.com/manual/core/aggregation-pipeline/>aggregation pipeline</a> stage <a href=https://docs.mongodb.com/manual/reference/operator/aggregation/geoNear/#pipe._S_geoNear><code>$geoNear</code></a> which returns an ordered stream of documents based on the proximity to a geospatial point. The output documents include an additional distance field.<p>The <code>near</code> field is mandatory and is the point for which to find the closest documents. In this field it is specified the type of geometry and the coordinates (given by the HTTP request).<p>The <code>distanceField</code> field is also mandatory and is the output field that will contain the calculated distance. In this case we’ve just called it <code>distance</code>.<p>Some other fields are <code>maxDistance</code> that indicates the maximum allowed distance from the center of the point, <code>minDistance</code> for the minimum distance, and <code>spherical</code> which tells MongoDB how to calculate the distance between two points.<p>We specify the maximum distance as the radio selected by the user in the frontend.<h2 id=frontend>Frontend</h2><p>As said earlier, our frontend will use Svelte. We already downloaded the template, so we can start developing. For some, this is the most fun part, because they can finally see and interact with some of the results. But for this interaction to work, we needed a functional backend which we now have!<h3 id=rest-queries>REST queries</h3><p>The frontend has to query the server to get any meaningful data to show on the page. The <a href=https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API>Fetch API</a> does not throw an exception if the server doesn’t respond with HTTP OK, but we would like one if things go wrong, so that we can handle them gracefully. The first we’ll do is define our own exception <a href=https://stackoverflow.com/a/27724419>which is not pretty</a>:<pre><code>function NetworkError(message, status) {
184 var instance = new Error(message);
185 instance.name = 'NetworkError';
186 instance.status = status;
187 Object.setPrototypeOf(instance, Object.getPrototypeOf(this));
188 if (Error.captureStackTrace) {
189 Error.captureStackTrace(instance, NetworkError);
190 }
191 return instance;
192}
193
194NetworkError.prototype = Object.create(Error.prototype, {
195 constructor: {
196 value: Error,
197 enumerable: false,
198 writable: true,
199 configurable: true
200 }
201});
202Object.setPrototypeOf(NetworkError, Error);
203</code></pre><p>But hey, now we have a proper and reusable <code>NetworkError</code>! Next, let’s make a proper and reusabe <code>query</code> function that deals with <code>fetch</code> for us:<pre><code>async function query(endpoint) {
204 const res = await fetch(endpoint, {
205 // if we ever use cookies, this is important
206 credentials: 'include'
207 });
208 if (res.ok) {
209 return await res.json();
210 } else {
211 throw new NetworkError(await res.text(), res.status);
212 }
213}
214</code></pre><p>At last, we can query our web server. The export here tells Svelte that this function should be visible to outer modules (public) as opposed to being private:<pre><code>export function get_area_info(long, lat, distance) {
215 return query(`/rest/get-area-info?long=${long}&lat=${lat}&distance=${distance}`);
216}
217</code></pre><p>The attentive reader will have noticed that <code>query</code> is <code>async</code>, but <code>get_area_info</code> is not. This is intentional, because we don’t need to <code>await</code> for anything inside of it. We can just return the <code>[Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)</code> that <code>query</code> created and let the caller <code>await</code> it as they see fit. The <code>await</code> here would have been redundant.<p>For those of you who don’t know what a JavaScript promise is, think of it as an object that represents «an eventual result». The result may not be there yet, but we promised it will be present in the future, and we can <code>await</code> for it. You can also find the same concept in other languages like Python under a different name, such as <a href=https://docs.python.org/3/library/asyncio-future.html#asyncio.Future><code>Future</code></a>.<h3 id=map-component>Map component</h3><p>In Svelte, we can define self-contained components that are issolated from the rest. This makes it really easy to create a modular application. Think of a Svelte component as your own HTML tag, which you can customize however you want, building upon the already-existing components HTML has to offer.<p>The main thing that our map needs to do is render the map as an image and overlay the selection area as the user hovers the map with their mouse. We could render the image in the canvas itself, but instead we’ll use the HTML <code><img></code> tag for that and put a transparent <code><canvas></code> on top with some CSS. This should make it cheaper and easier to render things on the canvas.<p>The <code>Map</code> component will thus render as the user moves the mouse over it, and produce an event when they click so that whatever component is using a <code>Map</code> knows that it was clicked. Here’s the final CSS and HTML:<pre><code><style>
218div {
219 position: relative;
220}
221canvas {
222 position: absolute;
223 left: 0;
224 top: 0;
225 cursor: crosshair;
226}
227</style>
228
229<div>
230 <img bind:this={img} on:load={handleLoad} {height} src="caceres-municipality.svg" alt="Cáceres (municipality)"/>
231 <canvas
232 bind:this={canvas}
233 on:mousemove={handleMove}
234 on:wheel={handleWheel}
235 on:mouseup={handleClick}/>
236</div>
237</code></pre><p>We hardcode a map source here, but ideally this would be provided by the server. The project is already complex enough, so we tried to avoid more complexity than necessary.<p>We bind the tags to some variables declared in the JavaScript code of the component, along with some functions and parameters to let the users of <code>Map</code> customize it just a little.<p>Here’s the gist of the JavaScript code:<pre><code><script>
238 import { createEventDispatcher, onMount } from 'svelte';
239
240 export let height = 200;
241
242 const dispatch = createEventDispatcher();
243
244 let img;
245 let canvas;
246
247 const LONG_WEST = -6.426881;
248 const LONG_EAST = -6.354143;
249 const LAT_NORTH = 39.500064;
250 const LAT_SOUTH = 39.443201;
251
252 let x = 0;
253 let y = 0;
254 let clickInfo = null; // [x, y, radius]
255 let radiusDelta = 0.005 * height;
256 let maxRadius = 0.2 * height;
257 let minRadius = 0.01 * height;
258 let radius = 0.05 * height;
259
260 function handleLoad() {
261 canvas.width = img.width;
262 canvas.height = img.height;
263 }
264
265 function handleMove(event) {
266 const { left, top } = this.getBoundingClientRect();
267 x = Math.round(event.clientX - left);
268 y = Math.round(event.clientY - top);
269 }
270
271 function handleWheel(event) {
272 if (event.deltaY < 0) {
273 if (radius < maxRadius) {
274 radius += radiusDelta;
275 }
276 } else {
277 if (radius > minRadius) {
278 radius -= radiusDelta;
279 }
280 }
281 event.preventDefault();
282 }
283
284 function handleClick(event) {
285 dispatch('click', {
286 // the real code here maps the x/y/radius values to the right range, here omitted
287 x: ...,
288 y: ...,
289 radius: ...,
290 });
291 }
292
293 onMount(() => {
294 const ctx = canvas.getContext('2d');
295 let frame;
296
297 (function loop() {
298 frame = requestAnimationFrame(loop);
299
300 // the real code renders mouse area/selection, here omitted for brevity
301 ...
302 }());
303
304 return () => {
305 cancelAnimationFrame(frame);
306 };
307 });
308</script>
309</code></pre><p>Let’s go through bit-by-bit:<ul><li>We define a few variables and constants for later use in the final code.<li>We define the handlers to react to mouse movement and clicks. On click, we dispatch an event to outer components.<li>We setup the render loop with animation frames, and cancel the current frame appropriatedly if the component disappears.</ul><h3 id=app-component>App component</h3><p>Time to put everything together! We wil include our function to make REST queries along with our <code>Map</code> component to render things on screen.<pre><code><script>
310 import Map from './Map.svelte';
311 import { get_area_info } from './rest.js'
312 let selection = null;
313 let area_info_promise = null;
314 function handleMapSelection(event) {
315 selection = event.detail;
316 area_info_promise = get_area_info(selection.x, selection.y, selection.radius);
317 }
318 function format_avg_noise(avg_noise) {
319 if (avg_noise === null) {
320 return '(no data)';
321 } else {
322 return `${avg_noise.toFixed(2)} dB`;
323 }
324 }
325</script>
326
327<div class="container-fluid">
328 <div class="row">
329 <div class="col-3" style="max-width: 300em;">
330 <div class="text-center">
331 <h1>Caceres Data Consultory</h1>
332 </div>
333 <Map height={400} on:click={handleMapSelection}/>
334 <div class="text-center mt-4">
335 {#if selection === null}
336 <p class="m-1 p-3 border border-bottom-0 bg-info text-white">Click on the map to select the area you wish to see details for.</p>
337 {:else}
338 <h2 class="bg-dark text-white">Selected area</h2>
339 <p><b>Coordinates:</b> ({selection.x}, {selection.y})</p>
340 <p><b>Radius:</b> {selection.radius} meters</p>
341 {/if}
342 </div>
343 </div>
344 <div class="col-sm-4">
345 <div class="row">
346 {#if area_info_promise !== null}
347 {#await area_info_promise}
348 <p>Fetching area information…</p>
349 {:then area_info}
350 <div class="col">
351 <div class="text-center">
352 <h2 class="m-1 bg-dark text-white">Area information</h2>
353 <ul class="list-unstyled">
354 <li>There are <b>{area_info.tree_count} trees </b> within the area</li>
355 <li>The <b>average noise</b> is <b>{format_avg_noise(area_info.avg_noise)}</b></li>
356 <li>There are <b>{area_info.census_count} persons </b> within the area</li>
357 </ul>
358 </div>
359 {#if area_info.trees_per_type.length > 0}
360 <div class="text-center">
361 <h2 class="m-1 bg-dark text-white">Tree count per type</h2>
362 </div>
363 <ul class="list-group">
364 {#each area_info.trees_per_type as [type, count]}
365 <li class="list-group-item">{type} <span class="badge badge-dark float-right">{count}</span></li>
366 {/each}
367 </ul>
368 {/if}
369 </div>
370 {:catch error}
371 <p>Failed to fetch area information: {error.message}</p>
372 {/await}
373 {/if}
374 </div>
375 </div>
376 </div>
377</div>
378</code></pre><ul><li>We import the <code>Map</code> component and REST function so we can use them.<li>We define a listener for the events that the <code>Map</code> produces. Such event will trigger a REST call to the server and save the result in a promise used later.<li>We’re using Bootstrap for the layout because it’s a lot easier. In the body we add our <code>Map</code> and another column to show the selection information.<li>We make use of Svelte’s <code>{#await}</code> to nicely notify the user when the call is being made, when it was successful, and when it failed. If it’s successful, we display the info.</ul><h2 id=results>Results</h2><p>Lo and behold, watch our application run!<p><video controls=controls src=sr-2020-04-14_09-28-25.mp4></video><p>In this video you can see our application running, but let’s describe what is happening in more detail.<p>When the application starts running (by opening it in your web browser of choice), you can see a map with the town of Cáceres. Then you, the user, can click to retrieve the information within the selected area.<p>It is important to note that one can make the selection area larger or smaller by trying to scroll up or down, respectively.<p>Once an area is selected, it is colored green in order to let the user know which area they have selected. Under the map, the selected coordinates and the radius (in meters) is also shown for the curious. At the right side the information concerning the selected area is shown, such as the number of trees, the average noise and the number of persons. If there are trees in the area, the application also displays the trees per type, sorted by the number of trees.<h2 id=download>Download</h2><p>We hope you enjoyed reading this post as much as we enjoyed writing it! Feel free to download the final project and play around with it. Maybe you can adapt it for even more interesting purposes!<p><em>download removed</em><p>To run the above code:<ol><li>Unzip the downloaded file.<li>Make a copy of <code>example-server-config.ini</code> and rename it to <code>server-config.ini</code>, then edit the file to suit your needs.<li>Run the server with <code>python -m server</code>.<li>Open <a href=http://localhost:9000>localhost:9000</a> in your web browser (or whatever port you chose) and enjoy!</ol></main><footer><div><p>Share your thoughts, or simply come hang with me <a href=https://t.me/LonamiWebs><img src=/img/telegram.svg alt=Telegram></a> <a href=mailto:totufals@hotmail.com><img src=/img/mail.svg alt=Mail></a></div></footer></article><p class=abyss>Glaze into the abyss… Oh hi there!