Programming Projects

MLND Project: Model Evaluation & Validation

10/02/2017 By debkr

3e Project: Model evaluation & validation

Project details

DESCRIPTION:

use Boston house price dataset to PREDICT selling price of a new/unseen home

PROCESS:

EXPLORE data > obtain important FEATURES & DESCRIPTIVE statistics abt data
Properly SPLIT dataset into TRAINING & TEST datasets
DETERMINE suitable PERFORMANCE METRIC for evaluating the problem
ANALYSE performance graphs for learning algorithm over varying TRAINING SET SIZES & with varying number of PARAMETERS
CHOOSE OPTIMAL MODEL that best generalises unseen data
TEST chosen optimal model on a NEW SAMPLE & COMPARE PREDICTED selling price to ACTUAL statistics

D3.js and Data Visualisation

11/07/2016 By debkr

Data analysis process:
When we encountered the data analysis process earlier in the year, we saw the basic process consists of: gather; clean; analyse (including, checking for accuracy); and finally, visualise/present. We’ve been doing lots of Python programming coupled with creating SQL databases to extract data from some source (web pages, files, XML or JSON files) and sort or store it in a database.

The process we’ve been using during the capstone course – and in line with the original Page/Brin search engine process – is to first collect the raw data and store it – unprocessed – into a holding database. From here we’ve gone on to clean up the data and save it in a more structured way in a new, relational database. This results in a smaller database which is quicker to search and retrieve data from. As I found when writing my own search engine application, these first two databases take a long time to retrieve the data, especially when the search engine’s reach is set widely. [Read more…] about D3.js and Data Visualisation

Simple Search Engine in Python

28/06/2016 By debkr

Part of the Python specialisation capstone (see Refs below) is to recreate a simple web search engine, modelled on the original Google search ranking algorithm (you can read the short version of Page and Brin’s 1998 Stanford paper here). The Google algorithm placed emphasis on information obtained from the HTML “link structure and link text” of all links found in all indexed web pages, and to use this information “for making relevance judgments and quality filtering”.

Google search algorithm:
The basic premise of the algorithm is a probability measure, expressed in laymen’s terms as: “how likely is it that a random surfer would alight on this particular web page if they just randomly surfed through all links on all pages on the web until they got bored and gave up”. The algorithm itself includes a measure of all incoming links to a web page (i.e. the number of “citations or backlinks” to that page), enhanced by the quality-ranking of each of those in-coming citation links. In this way, the search algorithm defines an objective page rank or search ranking for each web page. [Read more…] about Simple Search Engine in Python

Python + SQL: example database

05/06/2016 By debkr

XML + SQL + Python:
Here’s a quick example showing how powerful these elements are when we put them together – we can use Python to read data from an XML file, extract data elements we’re interested in, create an SQL database and upload the various data values into the database. We can then query and return various data selects direct from Python (although still have the option to view/query the database through the SQLite web browser as well).

Here I’ve taken the recipe XML data format and saved as a file, which looks like this: [Read more…] about Python + SQL: example database

Simple Tagging Engine (updated using tuples)

07/05/2016 By debkr

My simple Python Tagging Engine has now been updated to incorporate learnings about tuples, and incorporates simplified ways of sorting and returning counted values from a dictionary. Code and simple documentation available on GitHub: https://github.com/debkr/blog_tagger.

Ed Tech PTP: Summary Syllabus

03/05/2016 By debkr

My newly-established EdTech Personalised Training Plan includes the following streams:

1. Information Theory Stream
2. Learning/Development Stream: how humans learn vs. how machines learn + developing machines to augment human learning
3. Collaborative Learning Stream
4. Innovation Stream
5. Agile Methodologies Stream
6. Educational Technology Stream: to review the technological and presentational innovations shaping education and L&D

Refer my earlier post “Learning About Learning” for some background on this new/additional PTP.

Learning About Learning

03/05/2016 By debkr

As I’ve been going along in this mission of discovery that is my mid-life career-change search, I’ve come to realise just how important Learning & Development really is to me. It’s become a key area for me to look further into – especially in response to my wholly-positive experiences of MOOC’s so far and in relation to my budding interest in all things programming- and technology-led.

I’ve been thinking more and more about the ideas Accelerated Learning and Augmented Learning (specifically in relation to an ongoing developmental Programming Project). In brief, this is about how machine learning and artificial intelligence can be used to help and empower humans, rather than just be there to make rich people more money and put more people out of jobs and into long-term unemployment or underemployment. [Read more…] about Learning About Learning