"If we are to learn anything about the ultimate nature of species we must reduce the problem to the simplest terms and study a few easily recognized, well differentiated species" – Edgar Anderson, Annals of the Missouri Botanical Garden, 1928
Recently I've added a Python API to a C library for deep learning known as libdeep. The libdeep project was started in 2013 and is one of my more popular open source projects on Github. The core of it is based upon back-propagation code which I originally wrote in the early 1990s when computers were a lot less powerful than they are now, and since that time the fortunes of neural networks within the popular science imagination have waxed and waned.
Neural computing is about as old as digital computing. In 1943 Warren McCulloch and Walter Pitts proposed a neuron-like computing element. Alan Turing picked up on that and in the post-war period he writes to the cybernetics pioneer Ross Ashby suggesting that Ashby use the new Pilot ACE machine to simulate a neural network. By the late 1950s perceptrons are invented and then after some argumentation their design was refined with the addition of a sigmoidal activation function. About a decade ago better ways of training neural nets with many layers were developed by Geoffrey Hinton, and that's the so-called "deep learning" that we know of today.
It should be said that most neural network systems running on computers bear little or no resemblance to biological brains. Biological neurons do a lot more stuff besides adjusting synaptic strengths. Classical neural nets could be characterised as a variety of (more or less practical) machine learning system inspired by what neurons were believed to do by the standards of neurophysiological understanding in the mid 20th century and are situated within the now largely discredited psychological paradigm of behaviourism.
My own deep learning implementation isn't the same as Hinton's. I'm not using Restricted Boltzmann Machines, although I am using dropouts and unsupervised feature learning with autocoders. Adding a Python API makes the system available to a larger number of software developers and might also include educational uses.
So without further ado let's start with the simplest possible example: the XOR problem. In the 1970s XOR was a problem, because perceptrons couldn't learn that logic function and so the criticism was leveled that this was a strong indication that neural nets were not going to be able to rival more traditional symbolic logic. First we can create a data set in the form of a CSV file from the XOR truth table.
0,0,0 0,1,1 1,0,1 1,1,0 0,0,0 0,1,1 1,0,1 1,1,0 0,0,0 0,1,1 1,0,1
There's repetition here, but that's only to give the system chance to randomly pick 80% of the data for training and 20% for testing. Then we can use the following Python program:
#!/usr/bin/python import deeplearn noOfSamples = deeplearn.readCsvFile("xor.csv", 4, 1, , 0) deeplearn.setErrorThresholds([0.07, 0.007]) deeplearn.setLearningRate(0.2) deeplearn.setDropoutsPercent(0.001) deeplearn.setPlotTitle("Xor Training") deeplearn.setHistoryPlotInterval(50000) print "Training started." timeStep = 0 while (deeplearn.training() != 0): timeStep = timeStep + 1 print "Training Complete. Test performance is " +
str(deeplearn.getPerformance()) + "%"; deeplearn.save("xor.nn")
This loads the data and creates a neural net with 4 hidden units and a single hidden layer. If you run that it will save an image called training.png, which shows the training error over time and at the end it saves the trained neural net to a file called xor.nn. We can then test that the XOR logic function was successfully learned with the following program:
#!/usr/bin/python import deeplearn deeplearn.load("xor.nn") print("0,0"), if deeplearn.test([0.0,0.0]) > 0.5: print "1" else: print "0" print("1,0"), if deeplearn.test([1.0,0.0]) > 0.5: print "1" else: print "0" print("0,1"), if deeplearn.test([0.0,1.0]) > 0.5: print "1" else: print "0" print("1,1"), if deeplearn.test([1.0,1.0]) > 0.5: print "1" else: print "0"
And if all is well then on running this program you should see the XOR truth table. Of course, XOR is not the most exciting thing ever, and the above is only intended to illustrate how you can implement deep learning in a small enough number of lines of Python code that you could probably use it for teaching purposes. You can mess around with the learning rate and other parameters to see how that alters the training graph or the test performance.
Now let's tackle a more interesting problem. In 1935 the botanist Edgar Anderson was researching the origin of species. Contrary to the notion of branching under strict descendancy in reproductive isolation perhaps hybridization within suitably conducive habitats could itself give rise to new species with overlapping phenotypic characteristics. He measured the physical dimensions of Iris plants growing together in the same colony in the Mississippi delta region, which belonged to three species: Iris setosa, Iris versicolour and Iris virginica. The resulting data set was later published by the statistician Ronald Fisher and has become a well known machine learning classification problem. It can be found here: http://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
To use this as a libdeep data source we'll first need to tidy it up. libdeep currently doesn't understand text, only numbers, so we can replace the species names with class 0, class 1 and class 2.
sed -i 's/Iris-setosa/0/g' iris.data sed -i 's/Iris-versicolor/1/g' iris.data sed -i 's/Iris-virginica/2/g' iris.data
And then similarly to before a Python program can be written to learn the classifications.
#!/usr/bin/python import deeplearn no_of_species = 3 deeplearn.readCsvFile("iris.data", 16, 3, , no_of_species) deeplearn.setErrorThresholds([0.5, 0.5, 0.5, 1.0]) deeplearn.setLearningRate(0.1) deeplearn.setDropoutsPercent(0.01) deeplearn.setPlotTitle("Iris Species Classification Training") deeplearn.setHistoryPlotInterval(500000) timeStep = 0 while (deeplearn.training() != 0): timeStep = timeStep + 1 print "Training completed. Test performance is " +
str(deeplearn.getPerformance()) + "%"; deeplearn.save("iris.nn")
This creates a neural net with 16 hidden units and three hidden layers to classify three types of things. Because this is a harder problem it takes longer to train than the XOR example. When the training is done another program can be run to test that the learning worked.
#!/usr/bin/python import deeplearn species = ["Iris setosa", "Iris versicolor", "Iris virginica"] deeplearn.load("iris.nn") print "Expected: " + species deeplearn.test([5.44, 3.436, 1.667, 0.214]) print "Returned: " + species[deeplearn.getClass()] print "\nExpected: " + species deeplearn.test([6.14, 2.75, 4.04, 1.32]) print "Returned: " + species[deeplearn.getClass()] print "\nExpected: " + species deeplearn.test([6.71, 3.14, 5.92, 2.29]) print "Returned: " + species[deeplearn.getClass()]
The measurements supplied to the test function are different from any of the samples in the data set, so this determines whether the system is able to generalise rather than just learning specific training values. If the training went well then we'll see that the expected and returned species classifications are the same. Rather than being a "toy" problem this is closer to something which could be a practical application on a mobile device used by an ecologist conducting a field survey.
So this was an introduction to the new libdeep Python API. We've only dealt with classification problems on fully labeled data sets, and there's more work to be done to support time series and recurrent networks.
- Using STPA with software-intensive systems
- Codethink achieves ISO 26262 ASIL D Tool Certification
- RISC-V: running GNOME OS on SiFive hardware for the first time
- Automated Linux kernel testing
- Native compilation on Arm servers is so much faster now
- Higher quality of FOSS: How we are helping GNOME to improve their test pipeline
- RISC-V: A Small Hardware Project
- Why aligning with open source mainline is the way to go
- Build Meetup 2021: The BuildTeam Community Event
- A new approach to software safety
- Does the "Hypocrite Commits" incident prove that Linux is unsafe?
- ABI Stability in freedesktop-sdk
- Why your organisation needs to embrace working in the open-source ecosystem
- RISC-V User space access Oops
- Tracking Players at the Edge: An Overview
- What is Remote Asset API?
- Running a devroom at FOSDEM: Safety and Open Source
- Meet the codethings: Understanding BuildGrid and BuildBox with Beth White
- Streamlining Terraform configuration with Jsonnet
- Bloodlight: Designing a Heart Rate Sensor with STM32, LEDs and Photodiode
- Making the tech industry more inclusive for women
- Bloodlight Case Design: Lessons Learned
- Safety is a system property, not a software property
- RISC-V: Codethink's first research about the open instruction set
- Meet the Codethings: Safety-critical systems and the benefits of STPA with Shaun Mooney
- Why Project Managers are essential in an effective software consultancy
- FOSDEM 2021: Devroom for Safety and Open Source
- Meet the Codethings: Ben Dooks talks about Linux kernel and RISC-V
- Here we go 2021: 4 open source events for software engineers and project leaders
- Xmas Greetings from Codethink
- Call for Papers: FOSDEM 2021 Dev Room Safety and Open Source Software
- Building the abseil-hello Bazel project for a different architecture using a dynamically generated toolchain
- Advent of Code: programming puzzle challenges
- Improving performance on Interrogizer with the stm32
- Introducing Interrogizer: providing affordable troubleshooting
- Improving software security through input validation
- More time on top: My latest work improving Topplot
- Cycling around the world
- Orchestrating applications by (ab)using Ansible's Network XML Parser
- My experience of the MIT STAMP workshop 2020
- Red Hat announces new Flatpak Runtime for RHEL
- How to keep your staff healthy in lockdown
- Bloodlight: A Medical PPG Testbed
- Bringing Lorry into the 2020s
- How to use Tracecompass to analyse kernel traces from LTTng
- Fixing Rust's test suite on RISC-V
- The challenges behind electric vehicle infrastructure
- Investigating kernel user-space access
- Consuming BuildStream projects in Bazel: the bazelize plugin
- Improving RISC-V Linux support in Rust
- Creating a Build toolkit using the Remote Execution API
- Trusting software in a pandemic
- The Case For Open Source Software In The Medical Industry
- My experiences moving to remote working
- Impact of COVID-19 on the Medical Devices Industry
- COVID-19 (Coronavirus) and Codethink
- Codethink develops Open Source drivers for Microsoft Azure Sphere MediaTek MT3620
- Codethink partners with Wirepas
- Testing Bazel's Remote Execution API
- Passing the age of retirement: our work with Fortran and its compilers
- Sharing technical knowledge at Codethink
- Using the REAPI for Distributed Builds
- An Introduction to Remote Execution and Distributed Builds
- Gluing hardware and software: Board Support Packages (BSPs)
- Engineering's jack of all trades: an intro to FPGAs
- Bust out your pendrives: Debian 10 is out!
- Why you should attend local open source meet-ups
- Acceptance, strife, and progress in the LGBTIQ+ and open source communities
- Codethink helps York Instruments to deliver world-beating medical brain-scanner
- Codethink open sources part of staff onboarding - 'How To Git Going In FOSS'
- Getting into open source
- How to put GitOps to work for your software delivery
- Open Source Safety Requirements Analysis for Autonomous Vehicles based on STPA
- Codethink engineers develop custom debug solution for customer project
- Codethink contributes to CIP Super Long Term Kernel maintenance
- Codethink creates custom USB 3 switch to support customer's CI/CD pipeline requirements
- Codethink unlocks data analysis potential for British Cycling
- MIT Doctor delivers Manchester masterclass on innovative safety methodology
- Balance for Better: Women in Technology Codethink Interviews
- Introducing BuildGrid
- Full archive