What I learned at PyConUK 2017

Phew! What a weekend! I’m on the train, on the way back from PyConUK 2017 and it’s been such an experience. There’s been so much to take in, I’m just going to have a bit of a brain dump here and come back to it over the next few months. Hopefully you will find some of it useful too. Feel free to comment if I’ve missed something out, or got something wrong, or you have a link that I may have missed!

 

Education, Education, Education

There was a strong education track running throughout the conference, with lots of talks, workshops, and kids trying stuff out on Raspberry Pis and micro-bits. I hadn’t come across micro-bits before and they look awesome. Got to get one. There is plenty for developers to get involved with too.
Also awesome is 13yo Josh, who has developed Edublocks – an application to help students transition from block-based programming systems like scratch to Python. It’s on Github, so all contributions are welcome.

Kushal Das demonstrated you can impress women by talking about python. And Anwesha went on to bring Pyladies to India.

You can learn for free with the Open University – Learning to Code for Data Analysis is available on OpenLearn 24/7, whenever you are ready to learn. There is also a FutureLearn version, which only runs at scheduled times, but has full support and discussions. I may have got those the wrong way round, but you get the idea.

 

Data, Data, Everywhere, and Not a Drop to Drink

PyData also had a track of talks covering the challenges of working with and analysing data. I’ve included a few other talks here that probably aren’t in the official data track, but still cover similar ground.

I actually saw the Natural language with word vectors talk by mistake – I was intending to be somewhere else – but it turned out to be one of the most interesting talks of the weekend. Slides here.

Pandas is one of those libraries that everyone in the data community uses, but I’ve always struggled to get to grips with. Alexander Hendorf‘s talk on pandas indexes was great though, with lots of simple examples. There is a static version of the notebook too.

Tom Augspurger is handy for further pandas resources.

David Seddon’s talk on Database concurrency with Django was so packed, I was sitting on the floor and couldn’t see the slides. He was so clear and enthusiastic though, I could follow it easily. And now I know about database concurrency.

I also now know that Bokeh rhymes with Okay (and just this second learned the word bokeh comes from the Japanese word for the blurred region of a photo with a narrow depth of field). It’s one of those words I’ve seen written down, but never heard anyone say it before. Anyway, it is awesome, both for data visualisation and as a backend for your web application. I also found out about argo floats, which are pretty awesome.

Something close to my day job in natural hazard modelling was this talk on earthquake analysis. Lots of good tools are available, as seen in the slides [pdf], especially Earthquakes, Quake Feeds, Matplotlib Basemap, and obspy.

 

Other Cool Things

I went to a workshop on git low-level commands; these are known as plumbing commands (not to be confused with the high-level “porcelain” commands we already know – pull, diff, add, etc.). Some examples:

git hash-object -w <path>     [add a file to the object store]
git update-index --add <path> [add a file to the index]
git write-tree                [write current index to the tree]

These three commands are the equivalent to the porcelain git add command. And so on.
During the discussion, someone mentioned this blog post, which sums up git perfectly. If it was consistent, it would be so boring.

Another great talk was by the very enthusiastic Simon Davy, all about the WSGI app platform Talisker. It wraps your existing tools, providing setup, logging, standardised endpoints, etc. In particular, he talked about the benefits of logging to stderr, for example:

• works in dev
• handles multiple processes/threads
• is agnostic about paths/permissions
• OS does persistence and rotation
• avoids stdout buffering
• still want on disk logs

And a word of warning: if your error-reporting tool says you have no errors then it is broken.

James Campbell told us all about trading cryptocurrencies with python, using backtrader and ccxt. *THIS IS NOT FINANCIAL ADVICE* Apparently it’s a good time to get into the market as it is young, fast-moving, and so far has shown some good returns. *DEFINITELY NOT FINANCIAL ADVICE*

There was some good advice from Mark Smith on refactoring your code. Keep interface changes to a minimum, as every time you release a breaking change, it’s an opportunity for your users to switch to another library. Also, it’s a good idea to always use semantic versioning.

One useful tip from a talk on logging: instead of log.info(“Hello {}”.format(name)), you should log.info (“Hello %s”, name). I’ve always used the first version, so will try and use the second from now on.

 

Quick roundup of useful Tools

Hypothesis can find edge cases for your tests.
Ortools used by geotechnical engineers alongside scipy.
Bluedot for wireless communication with Raspberry pi, etc. See also lightning talk slides.
MyPy uses static typing to help check for errors.

 

The Ones That Got Away

With so many concurrent sessions, I didn’t get to see everything I wanted, and by the sounds of it, these talks were also really good. I’ll add links to the slides if I manage to get them.
Everyday Security Issues And How To Avoid Them. Abstract.

How Close Can I Get Amazon’s Alexa To Black Mirror’s Cookie. Abstract. Slides.

Lazy sequences working hard. Abstract. Slides.

Add Guis To Your Data Pipelines With Jupyter Widgets. Abstract.
No slides for this, as apparently the whole session was conducted in a Jupyter notebook. Good work!

Finding Bugs For Free: The Magic Of Code Analysis. Abstract.
No slides yet, but handy tool can be found at https://lgtm.com/

 

PyCon has a YouTube Channel

I haven’t looked yet, but it’s probably all on there somewhere.
https://www.youtube.com/channel/UChA9XP_feY1-1oSy2L7acog/videos

 

Quick Plug for the Python Software Foundation

“The mission of the Python Software Foundation is to promote, protect, and advance the Python programming language, and to support and facilitate the growth of a diverse and international community of Python programmers.”
python.org/psf

 

And lastly, the main thing I’ll be taking home from this weekend is that when giving a talk, you must INCREASE YOUR FONT SIZE!

The coolest digital archive in the world

If you’ve ever tried to recover some vital files from an old CD-ROM or floppy disk, you’ll know that digital media has a very short lifespan. This has been a problem for archivists for many years, but now a digital doomsday vault has opened in northern Norway.

‘Doomsday’ library joins seed vault in Arctic Norway

I’ve always said this obsession with all things digital is overblown, but I love this idea of converting digital media to film, so it can be stored indefinitely. Analog for the win, indeed.

Now we just need a vinyl library to store the world’s audio…

 

 

Calderdale Green Party submission to the Petitions Committee inquiry on grouse shooting

Calder Valley has been flooded four times in the last five years, partly due to unprecedented levels of rainfall, but partly due to land management practices on the moors above, as explained here.

By keeping the estates ready for grouse shooting, a vital flood defence is being lost, and no amount of walls and dredging downstream will be able to compensate.

If you live in the area, please contact your MP to try to end the practices of draining and burning and help the valley better cope when the next downpour comes.

Turning Calderdale Green

More people in Calder Valley signed Mark Avery’s petition to the government to ban driven grouse shooting, than in any other constituency.  Calderdale Green Party has sent a submission to the Petitions Committee inquiry on grouse shooting, to explain why.

If you live in Calder Valley, please let Craig Whittaker know that you expect him to attend the Parliamentary debate on the petition, and to stand up for his constituents. He has an open drop-in surgery in Hebden Bridge Methodist Church on Saturday 15th October, from 1.30pm. And there is Treesponsibility transport to London for the House of Commons debate on the petition on 18th October.

Labour MPs are starting to support a ban on driven grouse shooting.  If you live in Halifax and Holly Lynch is your MP, please use this info  about the Labour position on grouse shooting  and ask her to vote for a ban on grouse…

View original post 2,063 more words

It’s Such a Drag

So, in my day job as a GIS developer, I’ve been making an addin for ArcGIS. Nothing fancy, just a data loader to make it easier for our users to pick out the exact item they want from the many datasets we provide. Choose a few categories, select some items from the list, click the load button, done! But what if you want to just drag the items onto the map? ArcObjects doesn’t appear to offer drag and drop straight out of the box, but it is possible. If you’re also developing ArcGIS addins, I’ll save you some time and explain how it is done.

Implementing drag and drop in .NET is actually pretty straightforward. There are two parts to the transaction – the drag source (the control you are dragging from) and the drop target (the control you are dragging to). All you have to do is add an ItemDrag event to your source control (which calls DoDragDrop to start the drag action), then enable the AllowDrop property on your target control, and add the DragDrop event to handle the dropped item. There are some other events that can be handled, including DragEnter, DragLeave, DragOver, GiveFeedback, and QueryContinueDrag, but that is the basic procedure.

When developing addins for ArcGIS, the drop target is already handled for you – the main window in ArcMap is listening for drop events, such as dropping mxds or datasets from the Catalog window or from ArcCatalog. All you have to do is package your drag source into the correct format and you’re good to go. It couldn’t be easier, right?

Well, it all depends on being able to format the dragged item correctly. As this post from The Sandpit states, ArcMap accepts two kinds of input. When dragging from ArcCatalog, an ESRI Names object is created, which is an enumerator that provides the names of the datasets to be imported. Dragging from ArcMap on the other hand, creates an ESRI Layers object, which is a collection of layers and tables. Apparently setting the drag source to an ESRI Layers object is not supported, and in my addin we are just loading datasets from the database by name, so it looks like I need to use ESRI Names.

Incidentally, if you’re doing any development with clipboard objects, I found two helpful applications. Clipboard Viewer shows whatever is currently on the clipboard and DataObjectViewer shows whatever has just been dragged onto it. You can use either of these to drag a selection of layers or datasets from ArcMap or ArcCatalog to confirm which object type is being used.

So, the example code on the Sandpit post went some way towards constructing an ESRI Names object and setting it as the drag source, but it relied on a third-party dll which does not work on recent versions of ArcGIS. However, according to this Geonet post, you can instead use a MemoryStream to pass the IEnumName to a DataObject, then use the DataObject as the drag source.

I did this, and it worked! The dataset name dropped onto the map and was loaded as a layer. However, the debugger showed a log of exceptions with the message, “A first chance exception of type ‘System.Runtime.InteropServices.COMException’ occurred in System.Windows.Forms.dll”. This is not ideal – I always try to get my code to run as cleanly as possible – so I took a closer look at the DataObjectViewer page above, as it goes into a lot of detail about DataObjects and how to construct one from scratch. I managed to do this, but it didn’t make any difference unfortunately. The consensus among developers on Stack Overflow seems to be that first chance exceptions can be ignored as users don’t see them. Hey ho, as long as it works, right?

The full code is shown below. lvDatasets is the ListView that contains the dataset names to be dragged. To keep this example simple, it has been set up so all the items added to it are IDatasetNames. It may be more useful though, to create a custom class to hold the IDatasetName, the alias, and other related information.

    using System;
    using System.IO;
    using System.Windows.Forms;
    using ESRI.ArcGIS.Carto;
    using ESRI.ArcGIS.esriSystem;
    using ESRI.ArcGIS.Geodatabase;
    using DataObject = System.Windows.Forms.DataObject;

    private void LoadSelectedLayers()
    {
        // declare name objects to method scope and populate if dragging
        INameFactory nameFactory = null;
        IEnumName enumName = null;
        IEnumNameEdit enumNameEdit = null;
        DragDropEffects dropStatus = DragDropEffects.All;
        nameFactory = new NameFactoryClass();
        enumName = new NamesEnumeratorClass();
        enumNameEdit = (IEnumNameEdit)enumName;

        // iterate backwards over selected items so they are added in reverse order (so TOC order is same as list view order)
        for (int i = lvDatasets.SelectedItems.Count - 1; i >= 0; i--)
        {
            IDatasetName datasetName = (IDatasetName)lvDatasets.SelectedItems[i];
            IName tempName = (IName)datasetName;
            enumNameEdit.Add(tempName);
        }

        // Create a new data object.
        DataObject myDataObject = new DataObject();

        // Add the names to the DataObject.
        MemoryStream memoryStream = new MemoryStream((byte[])nameFactory.PackageNames(enumName));
        myDataObject.SetData("ESRI Names", memoryStream);

        // set the data as a drop source
        dropStatus = lvDatasets.DoDragDrop(myDataObject, DragDropEffects.All);

        if (dropStatus != DragDropEffects.None)
        {
            // reset list view
            ClearSelectedItems();
        }
    }

 

Water always wins

We get a lot of junk mail through the door, but I always have a look through the Valley Life magazine*, as between the small ads, recipes, and gardening tips there is usually one article featuring of old photographs of the area. Many of the scenes are still recognisable and I love trying to work out where the photographer was standing, which buildings are still there, which have been demolished, and so on.

This month’s issue delves into the fascinating world of civil engineering and Victorian architecture and describes how Widdop reservoir supplies Halifax with fresh water. The reservoir was built in the 1870s and holds 640,511,000 gallons of water, but lies high above Hebden Bridge, with nearly 8 miles (as the crow flies) of hills and valleys to cross before reaching the town. Work was supervised by the excellently-named John F La Trobe Bateman (FRSE FRS MICE FRGS FGS FSA and President of the Institution of Civil Engineers in Britain 1878-9), who seems to be responsible for constructing much of the country’s water supply infrastructure at the time.

A number of valleys had to be crossed en route to the city – in some places, aqueducts were built, but elsewhere an ingenious device called an inverted syphon was employed.

The principle is similar to the u-bend under a sink, and as we can see here, fairly simple to set up (that site gives me flashbacks to Chemical Engineering classes. So glad I didn’t spend the next three years calculating flows).

One of the syphons lies between Hollins Hall and Pecket Well, but the land on the other side of the valley is higher, so a tunnel had to be dug through the hillside. The Castle Carr tunnel is 2550 yards long and has three ventilation shafts, the deepest of which is 476 yards. They can be clearly seen on the OS 1:25,000 maps if you have one (if not Bing will be able to help you, just change the view to Ordnance Survey). The shafts are listed by Historic England, as they show “Good examples of stonemasons work on top of the moors in an isolated location”.

I’d noticed these shafts a few weeks ago when I was running up there, but couldn’t quite work out what they were. It’s too far from the railway line, so unlikely to be a train tunnel; I assumed it was a mine of some sort. It’s not a good place for mining though, as apparently the rock is so hard that the explosive charges were just blasted out of the drill holes, having no effect on the rock at all.

The tunnel ends at the Castle Carr estate, a country retreat which the article says contains the highest gravity-fed water feature in the UK (though the fountains at Chatsworth and Stanway are higher. Perhaps they meant height above sea level. At 300 metres it’s pretty hard to beat). I’d never heard of this place before and it’s not surprising – it has fallen into ruin and is only open to the public one day a year. This report makes it sound like it is really worth visiting, so I’ll try and make the trip next month.

Maybe I’ll get round to writing another post about it!

[*] No hyperlink, as the address they give, http://www.facebook.com/Valley-Life-magazine redirects to http://www.facebook.com/valleylifemagazine, which is an Italian lifestyle magazine. Not sure what’s going on there. I’m not on Facebook, so maybe that kind of thing happens all the time. Who knows?

Mapped: The whole of underground Manchester

I’ve always been fascinated by the hidden tunnels and chambers beneath our streets and Manchester has more than its fair share. I bought the book Underground Manchester a few years ago and thought it pretty comprehensively covered this secret world most of us know nothing about. However, a recent article in the Manchester Evening News shows that the city has many more stories to tell.

Mancunian enthusiast Mark Crossfield has spent the last few years putting together Hidden Manchester – a site dedicated to his love of the Manchester Underground. The MEN article contains a few good examples, but I thoroughly recommend a look at the site as it is packed with information.

Cartographically, it is not very exciting – just a push-pin overlay on a satellite basemap – but all the features have popup information with links for further reading. There isn’t any attribution on the basemap (I think it’s Google) and it doesn’t seem to be possible to switch to a vector map instead of imagery or turn on a street labelling layer either, but this map is definitely more about content than presentation. The sources page is full of interesting links and is probably worth the price of admission alone. There is certainly enough here to keep me quiet for some time.

 

Digitisation of Japanese Maps at the John Rylands Library

More from the excellent Rylands map collection.

Heritage Imaging Manchester

Digitised material is progressively being added to the Library’s imaging online collection – LUNA – It has grown to include another small but very important part of our Special Collections.

A number of Japanese Maps have recently been digitised with the support of the Library’s Digitisation Steering Group. The Japanese Collection, assembled by the 25th Earl of Crawford in the 1860s and 1870s and purchased by the John Rylands Library in 1901, is not large by international standards, but it contains a number of manuscripts and printed books of great interest and rarity. Amongst them are a number of 18th and 19th century maps together with topographical or geographical books and manuscripts.

Initiated by Erica Baffelli – Senior Lecturer in Japanese Studies, University of Manchester – The aim behind this project was to select and digitise a number of maps and associated books and manuscripts of the Library’s Japanese Collection in…

View original post 434 more words

Could not load file or assembly

So, I changed jobs recently* and have been mostly brushing off my C# skills to write ArcGIS Addins. This has been largely frustrating, as Esri and Microsoft have joined forces to make things as difficult as possible. For example:

 

“error : Could not load file or assembly xxxx or one of its dependencies”. Do you feel like telling me which one, Visual Studio? #headdesk

— Jon Morris (@Jon_Two) March 20, 2015

 

This error message was particularly annoying, as it is fairly ambiguous. The full text isn’t much more enlightening:

Could not load file or assembly xxxx or one of its dependencies. The system cannot find the file specified. C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\TeamTest\Microsoft.TeamTest.targets

If only there was a way for the system to tell my which dependency it was having a problem with. Never mind, I’ll go through them all one by one and get there eventually.

For the record, the culprit was ESRI.ArcGIS.Desktop.AddIns, and the solution is to change the CopyLocal flag to true. Makes sense that if the file isn’t copied locally, it can’t be found.

The question is, why was CopyLocal set to false in the first place? I’m sure it was true last time I looked. Not only that, but this bug has come back a few times. At least I know what the fix is this time. I still didn’t know why that flag kept changing, even though I’ve done many searches.

Finally today, I worked out what was going on – the CopyLocal flag is reset every time you update the file Config.esriaddinx. Why should it do this? Who knows, but at least I know the solution now, and I’ve written it down, so when I start searching for the answer in three months time, I’ll find it!

*Actually four months ago, but geologically speaking, that’s just a blink of an eye.

The Catawba Deerskin Map: A Rare Example of Native American Cartography

Continuing the Native American theme, here’s a map showing that sometimes people are more important than places.
Thanks to @globemakers for the tipoff!

Petros Jordan

This year I would like to explore more unusual methods of mapmaking by a more diverse group of creators.  Throughout history, maps by powerful nations and empires have proliferated, but maps by the less powerful actors, if they exist at all, are seldom seen.  In this way, we have come to see the world through the eyes of the conquerors, but almost never through the eyes of the conquered.

In the interest of opening our eyes to new perspectives, I want to share a unique Native American map I discovered while flipping through one of the map books I received for Christmas (yes, I got more than one).  This map was drawn up by a chieftain of the Catawba tribe that resided in what would become the Southeastern United States.  At the time of the map’s creation in 1721, though, this land was being colonized by the British.  The British settlements hugged the…

View original post 374 more words

Lost in the Infinite Wilderness of America

Did you hear about the Welsh-speaking tribe of American Indians? No? Well, indie rock genius Gruff Rhys is travelling the world, playing songs from his film, book, and album American Interior, which tells the tale of John Evans, the man who went to find them.

The story starts with the Norman invasion, when the Celts were pushed back to the fringes of Britain, with the Welsh territory largely reduced to Anglesey and Snowdonia. Owain Gwynedd strengthened the kingdom, but on his death, his successors began to fight amongst each other, causing his son Prince Madoc to decide to set sail in search of new lands to settle. Apparently, he made it all the way to America and established a settlement in Mobile, Alabama. After returning to Wales to gather more supporters, he travelled up the Mississippi, eventually stopping near North Dakota. Here, his descendants remained, passing on their language and skills such as building coracles.

That’s the legend anyway, and it became fairly popular in Elizabethan times, as Britain tried to claim more territory in the new lands of America by insisting that they had actually got there first. Some years later, Iolo Morganwg decided to raise an expedition to find this lost tribe and put Wales back on the world map. Unfortunately, there was only one applicant, John Evans.

Evans and Morganwg set off to London to find some benefactors for their trip, but after little success, Iolo scarpered back to Wales, leaving Evans penniless in the big city. He decided to go ahead and travel to the US anyway.

The inner sleeve of Rhys’ LP shows Evans’ route from Baltimore, then Philadelphia (“where they taught him to make maps”), and onwards to the Mississippi and finally to the lands of the Mandan in present day North Dakota.

[Map by Pete Fowler, who did most of Super Furry Animals’ artwork]

American Interior LP inner sleeve

 

More maps and illustrations were on display during the show – here’s Gruff playing in front of maps of the Ohio River (“it looked exactly like this”) and the Canadian border at the 49th parallel (I think you probably had to be there).

 

Evans had many adventures along the way including catching malaria, being imprisoned, defecting to the Spanish, persuading the English army to abandon a fort (thereby annexing the land for Spain), defending the Canadian border, and finally sailing the length of the Mississippi before dying in New Orleans. The whole saga is summed up excellently in the song 100 Unread Messages.

Rhys’ sleeve notes above (and detail below) highlight the difficulty of using the correct names when discussing the indigenous people of North America. History is written by the victors and too often, a name given by outsiders can stick, even though it causes offence.

Luckily for us, a young cartographer has produced a map of Native American tribes showing their names and locations before the Europeans arrived. Below is a detail of the map, showing the Mandan nation, the most likely candidates for the lost tribe that John Evans was searching for.

So, even though Evans never found the lost Welsh tribe, his efforts weren’t in vain, as his maps were used by Lewis and Clark during their journeys through the American west.

Mystery still seems to surround Evans though, as the above site also seems to think he is Scottish. Oh well, as I’ve said before, print the legend.