DaveParr.Info

Moving Starpilot to GraphQL

2024-01-16T00:00:00+00:00

After building Starpilot</a> at the end of last year and using it for a little. I became pretty frustrated with the time it took to create the vectorstore. Specifically, the time it took to read the relevant repo data from GitHub’s REST api via PyGithub</a>. I outlined the choices I made</a> in the original blog post, but to summarise: PyGithub</code> wraps the GitHub REST api and is used to read the repos that are starred by a user on GitHub, and then pass this to a function that iterates over each</em> repo for each</em> piece of data I want to extract. I’m sure that for many (most) use cases this isn’t an issue, but for getting 9 pieces of data for lists of sometimes hundreds or even thousands of repos, it’s the wrong tool for the job. There is overhead to set up the connection, then execute the call, and wait for the response to come back. The total number of calls is roughly:</p>

9[pieces of information] * 1000[number of repos] + 1[list of user starred repos]
</span></code></pre>
This is slow. To be fair I wasn’t surprised by this. I actually expected it and just accepted it so I could get onto making the tool somewhat useful. Now I know it is useful though, it’s time to optimise. For my list of 800 starred repos it takes about 20 minutes on my desktop. Luckily, I’ve found a way to 10x the speed of this process down to 2 minutes for the same list on the same machine using GraphQL.</p>
How I did it</h2>
GraphQL</h3>
GraphQL</a> is a query language for APIs and a runtime for fulfilling those queries with your existing data. It provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools. GraphQL is perfect for the needs of this project as:</p>
GitHub GraphQL Explorer</h3>
GitHub provides a GraphQL api</a>. The Github GraphQL api allows you to structure a query for a specific set of information for a repo, then use that structure to return the same set of of structured information for each repo that a specific user has starred, all in one call. GitHub also provides a GraphQL explorer</a> to help you build your queries. However after a while I found this limiting, mostly because the page renders so that only around a 3rd of the screen is the playground with a mad amount of space for the header, footer and sidebar. Therefore I installed:</p>
Insomnia</h3>
Insomnia</a> is a desktop editor for developing APIs. I liked it because the workspace is more fully featured, allowing you to save queries and sync them to your cloud account for storage, and handles authentication and suppliying variables. Because part of the GraphQL specification is that each API can be introspected</em></a> it also behaved exactly like the GitHub Explorer and provided autocomplete, documentation and error linting for the queries I was writing. This was a huge help in getting the queries right. GitHub’s GraphQL interface is HUUUUGE. With Insomnia I was able to build up the query I needed in stages, testing each part as I went (Insomnia the app, not the sleep problem). However, I still needed to integrate this into my code. Most tutorials seem to suggest that the thing to do is just take the query, wrap it in """ myQuery here"""</code> and pass it to requests, but that seemed like a perfect way to accidentally break 90 lines of code with an unlintable typo. I wanted to use a python library that would allow me to build the query in a more structured way that is harder to break by accident, and also allow me to pass variables to the query. I found:</p>
graphql-query</code></h3>
denisart/graphql-query</a> is a Python package to build GraphQL queries from the core abstract pieces of the GraphQL language. I found this really useful as I learnt more about GraphQL and how it’s mental model is formed of fields with potentially optional arguments, nodes, edges and fragments. It also helped to make editing in VS Code directly a lot easier as it supports type hinting, so you can’t put an Argument</code> or Field</code> in the wrong place. As I iterated over the last tweaks to the query I found I actually didn’t need to go back to Insomnia as much. However, graphql-query</code> doesn’t actually act as a client to the GraphQL api, it just builds the query</a>. For that I needed:</p>
gql</code></h3>
gql</a> is a Python library for handling GraphQL queries in Python through a client. This became particularly useful as a (reasonable) limitation of GitHub’s GraphQL API is that when returning back a repo</code> type node, you can only return up to 100 per call. Therefore I needed to paginate through the results. I would have liked it if gql</code> had a way to handle this for me, but I couldn’t find it if it does, so I wrote a small bit of simple logic to handle that. The slightly tricky thing about pagination on this node is that you need to handle it with a GraphQL cursor</code>. This is a string that is returned with each node that you can use to tell the API where to start the next page of results from. I found that the easiest way to handle this was to use a while</code> loop</a> and pass the cursor back to the query as a variable. When the query return is empty for a last cursor, the loop breaks, and the results are returned. I did debate a more complex approach that involved counting the number of expected repos first, modulo 100, and then using that to determine the number of pages to request, but I decided that was overkill for my use case.</p>
Other tweaks</h2>
Because the data that is returned from the API comes as a dict for each repo, I modified the old function that iterated over each repo to get all the data for a specific repo. This function also contained the logic to write the data to disk for iteration over by langchain</code>, so it seemed a suitable change. As I’d also learnt some more about jq</code> and how it is used in langchain</code> I realised that instead of creating a json</code> file with 2 dicts, one for metadata {...}</code> and one for content {...}</code> I realised I could just write the data structure as a single dict</a>, and then use jq</code> to extract the data I needed</a> via tha optional metadata_func</code> argument in langchain.document_loaders.JSONLoader</code>. I got tripped up for a while as I had mistakenly written a behaviour into metadata_func</code> that effectively re-imputed</em> a None</code> value into the description</code> value after</em> the format_repo()</code> function had taken the value and</em> key out of the file saved to disk, which was then passed to Chroma</code> via the JSONLoader</code>. I managed to debug that however, and then I implemented the suggestion from the helpful error message to use langchain.vectorstores.utils.filter_complex_metadata()</code></a>  as a (slightly redundant) safeguard. I’d already got it wrong once so…</p>
The tweaks that didn’t make it</h2>
A version of the GraphQL query that I used originally went the whole hog and also returned the README.md/README.rst</code> from the repo. I’m certain that many of the readmes have great context that will enhance the embedding in the vectorstore, however I found that returning all that data significantly eroded the performance gains. To the extent that though I had 10x speeded up the process of reading the data from GitHub, I was now spending 10x longer generating the embedding via langchain.embeddings.GPT4ALLEmbeddings</code>. I know that the OpenAI embedding model seems to be much faster from some trials I’ve done, but I’m not (yet) happy to migrate over as it increases the cost of running starpilot</code>. However, not to the extent that it’s unviable (probably still less than a cup of coffee), but I’m not sure I’m ready to commit to that yet. Maybe as I work towards building an agent architecture for starpilot</code> I’ll revisit this. When I do, I’ll probably implement a way to summarise the use case from the readmes and strip out the things like installation instructions. That summary would then be the part used in the embedding.</p>
Conclusion</h2>
Using Github’s GraphQL API has allowed me to 10x the speed of reading the data from GitHub</a>. Developing for GraphQL is maybe even less</em> of a hassle than for REST APIs, as the introspection allows you to build complex queries in a more structured way with a better linting and autocomplete driven experience up front. I’m not sure that means all</em> API’s should be GraphQL. It’s pretty clear the benefits come when the data that is being returned (or operations being performed) are being done on data objects with a huge amount of structure. The other benefit is being able to batch up calls on that structure. The introspection aspects of it also help with learning, developer experience and good documentation, however arguably a chunk of this idea is already available on REST APIs via OpenAPI/Swagger, so it’s not a unique selling point.</p>
Developing GraphQL in Python also seems pretty robust, with a number of other options I didn’t include for GraphQL clients in the language ecosystem, as well as other tools that I didn’t need for starpilot like GraphQL servers. I had worried it would be pretty under tooled, and that GraphQL client side was a very “JavaScript” thing to do. Luckily that wasn’t the case.</p>
In the future either on this project or another I’d like to explore graphql-query</code> ’s support for fragments to dynamically extend the query render at call time (e.g. to include the readme data in the query based on a boolean flag in the function signature). I’d also maybe look at gql</code>’s support for creating a Domain Specific Language (DSL) for a given GraphQL schema</a>. I only spotted that after I had finished my work with graphql-query</code> so I didn’t spend time resolving the same problem, but if I need to write more GraphQL in Python I’ll definitely look at that.</p>
Hopefully this speed up will make starpilot</code></a> more approachable for a lot of people. I know I’ve been irritated in my own development cycle iterating over star pilot read</code> and waiting for the data to come back, so at the very least I’ve saved myself some time and learnt a bunch in the process.</p>



Copilot for your GitHub stars
2023-11-19T00:00:00+00:00
How do you use your GitHub stars? </p>
I’d guess if you’ve been programming for a few years you’ve probably hit the star button at the top of a few of your favourite repos. I know some people I follow have done it thousands</em> of times. Do you go back to them though? Do you review them for inspiration for your next project or go to them when you’re stuck on a partictular problem?</p>
Inspiration</h2>
I’ve always assumed I would use them but I never have. I found myself doing some research recently into how to build software that uses LLMs, with the deliberate goal of building an as yet undefined side-project. I wanted to build something I hadn’t built before, something that was hopefully a little original, and maybe even useful</strong>! So yet again I was starring repos like LangChain and Chroma, swearing this time would be different. </p>
As I was running through blog posts and diligently smashing the star buttons I realised that I had just hit on exactly what I wanted to try. I wanted to bring my GitHub stars right into my editor. I wanted to be able to have them next to me as I was working and get a sensible set of suggestions on what might be useful for my needs at that moment, and I had just been starring the exact repos that could make this happen!</p>
The original idea</h2>

Use a dataset of your personal stars to inform retrival augmented generation for a question and answer large language model deployed in a command line interface</p>
</blockquote>
I thought this would be useful for a few reasons:</p>

By having it in the CLI its available right in my editor, and to every project.</li>
By having a set of your personal stars the suggestions are already curated by your interests and preferences. Mine are all Python and R librarys, wierd data bases and charting libraries. Yours might mostly be Ruby gems, or web frameworks, or tools for embedded systems.</li>
By using a large language model the tool might</em> be more capable of understanding the intentention of your goals, for instance the query “Suggest how to build a web app” might be able to infer that you’d likely want a front end component, a backend component and a data storage component, and might even deal with servers and deployment.</li>
By using large language models the tool might</em> be more capable of semantic search rather than keyword matching which suits this problem as there is no strong standard on how a library describes it self through it’s topics, description and documentation.</li>
</ul>
Semantic vs keyword</h3>
Keyword</h4>
A keyword search looks for the exact letters in a string, or potentially a partial match. As an example the query "Data Science"</code> would find things that exactly matched the charcters in the string "Data Science"</code> and maybe also ["Data", "Science", "DS"]</code>.</p>
Semantic</h4>
A semantic search looks for the conceptual</em> similarity between things, so in this context "Data Science"</code> would find things that matched the vector embedding of "Data Science"</code> as well as maybe also the vector embeddings that are associated with ["Machine Learning", "Artificial Intelligence"]</code></p>
And so I ran poetry new starpilot</code>:</p>
starpilot</a></p>
Why retrival augmented generation matters</h2>
Retrieval Augmented Generation (RAG) is a technique used by large language models to cope with some of the limitations inherent in what are also sometimes referred to as ‘Foundational’ models. </p>
When a model like GPT3 is trained, it is fed large amounts of textual data written by humans. These get translated into ‘weights’ in a nueral net. To overly simplify, these weights tell the model what the next most likely text is that follows the text it has already been shown. </p>
However, these models don’t know much about what has happened recently, what other programming resources really exist rather than what just sounds like it should exist, or where to exactly get a specific repo or webpage.</p>
Retrieval augmented generation solves this by allowing you to feed the large language model with known real, up to date and relevant information. </p>
Vectorstores</h2>
A type of data base called a vectorstore is commonly used for this because they are deliberately optimised towards a similarity search use case. They achieve this in a few ways:</p>

Vectorstores store what you pass them as a ‘vector embedding’. A vector embedding takes data (like text or images) and converts them to a list like representation of numbers.</li>
Vectorstores keep similar vector embeddings close together in memory. This means that they are as fast as possible at returning lots of documents that have similar semantic meaning, because they are all clustered together.</li>
Vectorstores have APIs that are specifically designed for these use cases, with querying methods that lean towards semantic searches more than sql queries, and loading techniques that integrate tightly into other systems that generate these vector embeddings from large language models.</li>
</ul>
Designing a system</h2>
With this set of goals and new knowledge I got to work working out which puzzle pieces I needed and how to fit them together. This time I did go through my stars (and a few other things), though maybe this is for the last time!</p>
I figured I could get started using 4 main open source repos. My first commit to my pyproject.toml</a> used these projects:</p>
Typer</h3>
tiangolo/typer</a></p>
typer</code> is a pretty trendy framework for building CLI tools in python right now. It embraces typing, uses function decorators to magically turn your functions into CLI commands, and has relatively clear documention.</p>
I chose typer</code> specifically because:</p>

I wanted to see what the hype was about</li>
I think typing helps write better code</li>
I found the documentation really helpful to get started easily</li>
</ul>
langchain-ai/langchain</a></p>
langchain</code> is the most mature and well embraced large language model orchestration framework. Langchain itself doesn’t supply you with any specific llm or vector store or embedding approach. Instead it is deliberately ‘vendor agnostic’. It provides a common set of APIs and abstractions across a staggering number of vector data bases, large language models and embedding engines. </p>
I chose langchain</code> because:</p>

It is the most established tool in a brand new space</li>
I wasn’t really sure which suppliers of vectorstores and large languge models made the most sense for my use case</li>
I found the documentation really helpful to get started</li>
</ul>
Chroma</h3>
chroma-core/chroma</a></p>
chroma</code> is a vectorstore that has great support from Langchain. There are many others as well but Chroma won out at this stage because:</p>

I can run chroma</code> as an ‘embedded’ data store, e.g. it runs locally on the users machine</li>
chroma</code> was the most often used vectorstore in the Langchain docs for RAG tasks</li>
It was trivially easy to set up to the point at which I was convinced reading the tutorials that they had to have made a mistake</li>
</ul>
GPT4All</h3>
nomic-ai/gpt4all</a></p>
gpt4all</code> provides a set of LLM models and embedding engines that are also well supported by Langchain. gpt4all</code> was appealing because:</p>

It runs locally on ‘normal’ machines</li>
It seems well supported and maintained</li>
It is open about what data it was trained on and what data it will use to train on</li>
</ul>
PyGithub</h3>
Pygithub/PyGithub</a></p>
Soon after this I realised that pygithub</code> would be an easy way to go to GitHub to get the information I needed and bring it back into starpilot</code> to load into the vectorstore. I had initially thought I might be able to use the GitHub Document Loader</a> built into langchain</code>, though once I sat down to really work it out I realised that this doesn’t give access to a users stars, so I needed an alternative.</p>
The other way to build</h2>
There were alternatives in all these choices. I think these are all totally viable parts to build effectively the same system:</p>
Click</h3>
pallets/click</a></p>
I actually am using click</code>, sort of. typer</code> is built ontop of click</code>, but to be honest I didn’t really know that before I’d mostly decided. click</code> looks like a really great project, but it wasn’t as</em> clear how to get started.</p>
Llama Index</h3>
run-llama/llama_index</a></p>
llama_index</code> is probably a great project, but I only found it late in my thinking on this project. If I start a different project it’s suitable for any time soon I’m definately going to try it out as a comparison.</p>
Faiss</h3>
facebookresearch/faiss</a></p>
I’d used faiss</code> in a tutorial on vectorstores before. It didn’t strike me as hugely intuitive to use or as simple to set up (it’s recommended installation path is via conda). I also don’t particularly like Facebook so I’m happy to use an alternative. </p>
OpenAI</h3>
openai/openai-python</a></p>
I’d used openai</code> for a handful of tutorials and notebook experiments already and been very happy with it. However for a project like this I wasn’t really sure what the operational costs would be, and if they would be worth it for the benefit the tool provides. That combined with the requirement to have network connectivity while using the tool pushed me towards experimenting with alternatives. Luckily with langchain</code> I should be able to provide it as an optional backend in the future?</p>
What state is starpilot</code> now?</h2>
“actively developed”, “v0.1.0”, “untested” and “it runs on my machine” are good descriptions of the project right now.</p>
I’ve spent a few evenings this month on it, and see myself at least spending a few more on it next month. The API is getting breaking changes almost everytime I open the project. It’s got 0 real tests. It should get some soon though. It requires a few manual installation steps that are documented in README.md</code> but haven’t yet even been attempted on another machine other than the one I’m on right now.</p>
It also doesn’t yet achieve exactly what I want it to, but I see no reason yet that it can’t with some more development time.</p>
Current features</h3>
starpilot read MyCoolUserName</code></h4>
This will connect to Github and read the starred repos of the user MyCoolUserName</code>. Then it will go to each of those repos and get the topics and descriptions (and optionally the readmes) and load these into chroma</code> which is persisted on the local hard drive.</p>
starpilot shoot "insert topic here"</code></h4>
This will spin up the chroma</code> database and perform a semantic similarity search on the string given in the command, then return the documents that seem to be the most relevant.</p>
starpilot fortuneteller "Insert a question here"</code></h4>
This will perform the exact same search as the shoot</code> command, but then spin up a large language model and pass the results into the large language model for processing. It then returns the documents it found as well as the response from the LLM</p>
So….</h2>
That’s where this project is at. I’ve learnt a tonne about the available tools and relevant techniques in this space already, which was really the main goal of starting to begin with!</p>
That said the progress I’ve made so far only makes me more curious about what else can be done with this and what else can be solved towards the vision of “Making your GitHub stars more valuable in your daily coding”. Here’s some ideas that I’ve found exciting while getting my hands dirty that might show up in the future. These are along with the obvious things like any testing at all, a simpler way to set up the project on your machine, better error handling, a more sensible way to update the vectorstore than drop everything and rebuild each time, etc. </p>

Inspecting the current projects description (both it’s loose goals as well as more specific things like what packages it already uses) so that things that are already used aren’t suggested and are instead used to inform the response.</li>
Dynamically creating a GitHub list of similar starred repos for your user (though that would probably rely on this suggestion to extend the GitHub API</a>) so that you naturally have some ways of saving and sharing your starred repos that solve a specific problem between sessions in your terminal</li>
Building starpilot into a research agent that can perform actions such as installing the selected suggestion into the current project or be sent to GitHub to find new projects that solve the current goals that you haven’t starred yet</li>
</ul>
What do you think?</h2>
Does this sound like something intersting to you, maybe even something useful? Did this just spark inspiration in you for a new project? Does this actually already exist somewhere and I’m just being an idiot? Let me know :)</p>


My GitHub profile shows my popular DEV.to posts and GitHub repos automatically
2020-08-05T00:00:00+00:00
This is my github profile:</p>
DaveParr/DaveParr</a></p>
Background</h2>
I made it because I saw this repo by zhiiyang</p>
zhiiiyang/zhiiiyang</a></p>
Thanks to @mokkapps for this article which uses the twitter updating
action for discovering it.</p>
How I built a self updating readme</a></p>
Motivation</h2>
It took my a little while to decide what to do with my profile. I wanted
something data driven, and I wanted something to show-off my programming
successes. So I decided to show my most popular repos, and also to show
my most popular dev.to posts! I noticed that zhiiiyang made extensive
use of r-lib/actions</code></a>, and I’ve
found that it was really valuable for my project too!</p>
Method</h2>
Repos</h3>
The repos script was where I started. Building from zhiiiyang’s work, I
built a GitHub
workflow</a>
to call a script I had written.</p>
The script was quite
simple</a>.
It gets my repos from the GitHub API through the
gh</code></a> package, and tidies the return using
hoist</code> to grab the important bits. It then filters and pivots the data
into a simple plot.</p>
The most interesting part was where zhiiiyang added the output as a
commit by the action itself. The authentication for the action is
actually allowed by this section:</p>
env</span></span>:</span>
    GITHUB_PAT</span></span>:</span> ${{ secrets.GITHUB_TOKEN }}</span>
</span></code></pre>
I did stumble a little over the correct path to the plot image. It turns
out that you can use the relative location (./graph.png</code>) which will
work if you are viewing the README from within</em> the repo, but to make
it work when the README is displayed from my user</em> page you have to use
the absolute path
(https://github.com/daveparr/daveparr/blob/main/graph.png</code>).</p>
Posts</h3>
The posts was actually pretty easy once I’d got used to how GitHub
Action operate, and also how the GitHub README</code> profiles worked.</p>
The key part was actually a feature I recently developed for my own
package.</p>
DaveParr/dev.to.ol</a></p>
dev.to.ol</code> is a package to help R users manage their dev.to content. In
particular I had recently finished the functions that return data from
the API about your published articles. The key to this is to have the
DEV.TO api key that you want to use set as an encrypted secret in the
repo. once that is set, my package can read it if it’s set as an
environmental variable along with the GITHUB_PAT</code>.</p>
env</span></span>:</span>
    DEVTO</span></span>:</span> ${{ secrets.DEVTO }}</span>
    GITHUB_PAT</span></span>:</span> ${{ secrets.GITHUB_TOKEN }}</span>
</span></code></pre>
Because my package returns a tidy data.frame like object, it was trivial
to munge it down to just what I wanted to show, and then format it
neatly with knitr</code>. I also went all in on the r-lib/actions</code> examples,
and now not only generate the new data for both the chart and blogs
during the GitHub action, but also do the full compile from .Rmd</code> to
.md</code> using the setup-pandoc</code>
action</a>.</p>
Lessons in automation</h2>
I’d been meaning to explore GitHub Actions for a little while, and I
found a few things out that I’m going to be considering in the future as
I develop this an other projects.</p>
Pricing vs performance on macOS and Linux</h3>
It’s free to a point, and then you need to pay. I had a look at the
pricing
plan</a>
and noticed that your runtime impacts your pricing. Most of the examples
from r-lib/actions</code> run from macos-latest, as does zhiiiyang’s project.
In GitHub Actions pricing a minute of macOS runtime is worth 10
minutes</em> of linux runtime. I ran on macOS for a while too, but
eventually thought that it might be a smart idea for a long running
personal project to convert to a linux run time, though now I’ve done it
I’m debating going back to mac.</p>
The ‘problem’ is that I did not write this process to be fast, or light.
It’s a silly hobby project to over-automate because I can. Therefore, on
Linux I am now actually compiling the packages on each installation
AND</em> installing libcurl for the api
calls</a>.
The cost savings I make from not picking the faster run time in this
case are approximately negated by the increase in actual run time. By
eye, that part of the job runs at about 10 minutes now, where as on mac
it was about 10 times faster as the pre-compiled binaries could just be
downloaded and would run ‘out of the box’. I’ll probably change it back
in a little while after I’ve checked how variable it can be.</p>
Potential solutions could include some form of caching (which I’ve heard
is maybe supported?) or running the action in my own docker image with
the pre-compiled, though TBH that sounds like work, and this is supposed
to be fun :P</p>
r-lib actions for the Rmd</h3>
I really like the idea of compiling the README.md</code> for a package from
the README.Rmd</code> we often use in R. I’ve often forgotten in my own work
to do that key step before a push, and having a relatively simple
automation backed into where my repos live will likely be something I
use in the future. The best part of this trick is that r-libs/actions</code>
does the most irritating part of ‘making Pandoc work’ for me. So I can
just profit!</p>
Successs!</h2>
I really liked hacking this out. I got to put my dev.to.ol</code> package to
another practical lesson and learn about GitHub Actions. Feel free to
re-create this on your profiles, either by grabbing bits or by just
lifting the whole thing. One of the reasons I built it the way I did is
so it could be relatively portable between users, and maybe solve a
problem for more than just me. So if you get this deployed on your
profile, or get stuck, I’d love to hear from you!</p>


How to calculate a Pokemons 'power level' using kmeans
2020-07-17T00:00:00+00:00
library</span></span>(</span>pokedex)</span></span>
library</span></span>(</span>tidyverse)</span></span>
library</span></span>(</span>tidymodels)</span></span>
library</span></span>(</span>showtext)</span></span>
font_add_google</span></span>(</span>"</span>Press Start 2P"</span></span>)</span></span>
showtext_auto</span></span>(</span>)</span></span>

theme_set</span></span>(</span>theme_pokedex</span></span>(</span>)</span></span>)</span></span>

knitr::</span>opts_chunk$</span>set</span></span>(</span>fig.showtext</span>=</span>TRUE</span>)</span></span>
</span></code></pre>
The Heros Journey</h2>
Pokemon games have a very familiar cycle. You start with one of 3
Pokemon. You adventure out with your new buddy, facing tougher and
tougher Pokemon, in greater variety. Many of your Pokemon evolve over
time, and eventually you find the end-game legendaries in a climactic
battle of titans!</p>
You can even see this story in the data. Here is the base_experience</code>
of all the Pokemon, identified by game generation. The base_experience</code>
is the basic amount of experience that is gained by the winner of a
battle from a specific species. e.g. If you beat</em> a Bulbasaur, your
Pokemon gains</em> experience based on a formula which uses Bulbasaurs
base_experience</code>. Because of that we can see it as a proxy value for
how powerful</em> a Pokemon is. If it’s more powerful, it will be harder to
beat, and so reward you with more experience when you win.</p>
pokemon %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> id,</span> y</span> =</span> base_experience,</span> colour</span> =</span> generation_id)</span></span>)</span></span> +</span>
  geom_point</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Base Experience for each Pokemon"</span></span>)</span></span>
</span></code></pre>
</p>
In each generation you can see a few attributes:</p>

power level - 3(ish) tiers, grouped in different vertical lanes</li>
progression - a (generally) increasing trend in the base power value</li>
the up-ticks at the start and the end of each gen are the starters
top-tier evolution (Woo Blastoise!) and the Legendaries at the
end-game</li>
</ul>
Lets see if we can group each Pokemon into a power_level</code>. A
categorical grouping which relates it to other Pokemon with similar
base_experience</code></p>
Grouping and counting</h2>
Maybe we can explicitly describe the power levels of each tier of
Pokemon with a simple process? Can we group each Pokemon by evolutionary
chain, and then count each Pokemons order within the group?</p>
pokemon %>%</span>
  group_by</span></span>(</span>evolution_chain_id)</span></span> %>%</span>
  mutate</span></span>(</span>power_level</span> =</span> row_number</span></span>(</span>)</span></span>)</span></span> -></span> pokemon_group_count

ggplot</span></span>(</span>pokemon_group_count,</span>
       aes</span></span>(</span>
    x</span> =</span> id,</span>
    y</span> =</span> base_experience,</span>
    colour</span> =</span> as_factor</span></span>(</span>power_level)</span></span>
  )</span></span>)</span></span> +</span>
  geom_point</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Power level by group count"</span></span>,</span>
       colour</span> =</span> "</span>power_level"</span></span>)</span></span>
</span></code></pre>
</p>
Sort of? Generally we capture the pattern, but we don’t actually get it
very right. First off, there are more than 3 evolutionary tiers in our
engineered feature. We can also see there are some Pokemon are
classified as a power_level</code> higher than 1, but still in the lowest
group on this list. Why might have caused this?</p>
pokemon_group_count %>%</span>
  filter</span></span>(</span>power_level ></span> 3</span>)</span></span> %>%</span>
  pull</span></span>(</span>evolution_chain_id)</span></span> -></span> pokemon_group_count_mistakes

pokemon_group_count %>%</span>
  filter</span></span>(</span>evolution_chain_id %in%</span> pokemon_group_count_mistakes)</span></span> %>%</span>
  select</span></span>(</span>id,</span>
         name,</span>
         base_experience,</span>
         generation_id,</span>
         evolution_chain_id,</span>
         power_level)</span></span>  %>%</span>
  arrange</span></span>(</span>evolution_chain_id)</span></span> %>%</span>
  select</span></span>(</span>id,</span> name,</span> base_experience,</span> evolution_chain_id,</span> power_level)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
id</th> name</th> base_experience</th> evolution_chain_id</th> power_level</th></tr></thead>

43</td> Oddish</td> 64</td> 18</td> 1</td></tr>
44</td> Gloom</td> 138</td> 18</td> 2</td></tr>
45</td> Vileplume</td> 221</td> 18</td> 3</td></tr>
182</td> Bellossom</td> 221</td> 18</td> 4</td></tr>
60</td> Poliwag</td> 60</td> 26</td> 1</td></tr>
61</td> Poliwhirl</td> 135</td> 26</td> 2</td></tr>
62</td> Poliwrath</td> 230</td> 26</td> 3</td></tr>
186</td> Politoed</td> 225</td> 26</td> 4</td></tr>
106</td> Hitmonlee</td> 159</td> 47</td> 1</td></tr>
107</td> Hitmonchan</td> 159</td> 47</td> 2</td></tr>
236</td> Tyrogue</td> 42</td> 47</td> 3</td></tr>
237</td> Hitmontop</td> 159</td> 47</td> 4</td></tr>
133</td> Eevee</td> 65</td> 67</td> 1</td></tr>
134</td> Vaporeon</td> 184</td> 67</td> 2</td></tr>
135</td> Jolteon</td> 184</td> 67</td> 3</td></tr>
136</td> Flareon</td> 184</td> 67</td> 4</td></tr>
196</td> Espeon</td> 184</td> 67</td> 5</td></tr>
197</td> Umbreon</td> 184</td> 67</td> 6</td></tr>
470</td> Leafeon</td> 184</td> 67</td> 7</td></tr>
471</td> Glaceon</td> 184</td> 67</td> 8</td></tr>
700</td> Sylveon</td> 184</td> 67</td> 9</td></tr>
265</td> Wurmple</td> 56</td> 135</td> 1</td></tr>
266</td> Silcoon</td> 72</td> 135</td> 2</td></tr>
267</td> Beautifly</td> 178</td> 135</td> 3</td></tr>
268</td> Cascoon</td> 72</td> 135</td> 4</td></tr>
269</td> Dustox</td> 173</td> 135</td> 5</td></tr>
280</td> Ralts</td> 40</td> 140</td> 1</td></tr>
281</td> Kirlia</td> 97</td> 140</td> 2</td></tr>
282</td> Gardevoir</td> 233</td> 140</td> 3</td></tr>
475</td> Gallade</td> 233</td> 140</td> 4</td></tr>
789</td> Cosmog</td> 40</td> 413</td> 1</td></tr>
790</td> Cosmoem</td> 140</td> 413</td> 2</td></tr>
791</td> Solgaleo</td> 306</td> 413</td> 3</td></tr>
792</td> Lunala</td> 306</td> 413</td> 4</td></tr>
</tbody></table>
So there are some clear problems with this approach. In Gen 1 we had
some branching evolution with the Eevee family. Not only was this family
expanded in multiple generations to eventually 8 variations, but we also
saw more branching evolution trees as well. We also got ‘baby’ Pokemon.
Pokemon that are actually pre-cursors to other Pokemon, but are listed
later in the Pokedex.</p>
Luckily there is another variable we can use, that should be a whole lot
better.</p>
evolves_from_species</code></h2>
Each Pokemon that evolves from another Pokemon has the Pokemon they
evolve froms id</code> as the value in the evolves_from_species_id</code>
variable. Maybe we can use that to break up the Pokemon into their
‘power levels’.</p>

The following code is not my best work, however I spent some time on a more recursive strategy, but it was honestly miles more confusing. For the purposes of a silly example for a blog, I think this is prefferable.</p>
</blockquote>
pokemon %>%</span>
  mutate</span></span>(</span>power_level</span> =</span> case_when</span></span>(</span>is.na</span></span>(</span>evolves_from_species_id)</span></span> ~</span> 1</span>)</span></span>)</span></span> %>%</span>
  filter</span></span>(</span>power_level ==</span> 1</span>)</span></span> -></span> pokemon_1

pokemon %>%</span>
  mutate</span></span>(</span>power_level</span> =</span> case_when</span></span>(</span>evolves_from_species_id %in%</span> pokemon_1$</span>id ~</span> 2</span>)</span></span>)</span></span> %>%</span>
  filter</span></span>(</span>power_level ==</span> 2</span>)</span></span> -></span> pokemon_2

pokemon %>%</span>
  mutate</span></span>(</span>power_level</span> =</span> case_when</span></span>(</span>evolves_from_species_id %in%</span> pokemon_2$</span>id ~</span> 3</span>)</span></span>)</span></span> %>%</span>
  filter</span></span>(</span>power_level ==</span> 3</span>)</span></span> -></span> pokemon_3

bind_rows</span></span>(</span>pokemon_1,</span> pokemon_2,</span> pokemon_3)</span></span> %>%</span>
  arrange</span></span>(</span>id)</span></span> %>%</span>
  mutate</span></span>(</span>power_level</span> =</span> as_factor</span></span>(</span>power_level)</span></span>)</span></span> -></span> pokemon_evolves_from

pokemon_evolves_from %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> id,</span> y</span> =</span> base_experience,</span> colour</span> =</span> power_level)</span></span>)</span></span> +</span> 
  geom_point</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Power level by evolves_from"</span></span>)</span></span>
</span></code></pre>
</p>
Hmm, that’s actually worse? Lets focus on the Pokemon labelled
power_level</code> 1, but are up where we would expect level 3 Pokemon to be.</p>
pokemon_evolves_from %>%</span> 
  filter</span></span>(</span>base_experience ></span> 200</span> &</span> power_level ==</span> 1</span>)</span></span> %>%</span> 
  select</span></span>(</span>id,</span> name,</span> base_experience)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
id</th> name</th> base_experience</th></tr></thead>

144</td> Articuno</td> 261</td></tr>
145</td> Zapdos</td> 261</td></tr>
146</td> Moltres</td> 261</td></tr>
150</td> Mewtwo</td> 306</td></tr>
151</td> Mew</td> 270</td></tr>
243</td> Raikou</td> 261</td></tr>
244</td> Entei</td> 261</td></tr>
245</td> Suicune</td> 261</td></tr>
249</td> Lugia</td> 306</td></tr>
250</td> Ho-Oh</td> 306</td></tr>
251</td> Celebi</td> 270</td></tr>
377</td> Regirock</td> 261</td></tr>
378</td> Regice</td> 261</td></tr>
379</td> Registeel</td> 261</td></tr>
380</td> Latias</td> 270</td></tr>
381</td> Latios</td> 270</td></tr>
382</td> Kyogre</td> 302</td></tr>
383</td> Groudon</td> 302</td></tr>
384</td> Rayquaza</td> 306</td></tr>
385</td> Jirachi</td> 270</td></tr>
386</td> Deoxys</td> 270</td></tr>
480</td> Uxie</td> 261</td></tr>
481</td> Mesprit</td> 261</td></tr>
482</td> Azelf</td> 261</td></tr>
483</td> Dialga</td> 306</td></tr>
484</td> Palkia</td> 306</td></tr>
485</td> Heatran</td> 270</td></tr>
486</td> Regigigas</td> 302</td></tr>
487</td> Giratina</td> 306</td></tr>
488</td> Cresselia</td> 270</td></tr>
489</td> Phione</td> 216</td></tr>
490</td> Manaphy</td> 270</td></tr>
491</td> Darkrai</td> 270</td></tr>
492</td> Shaymin</td> 270</td></tr>
493</td> Arceus</td> 324</td></tr>
494</td> Victini</td> 270</td></tr>
531</td> Audino</td> 390</td></tr>
638</td> Cobalion</td> 261</td></tr>
639</td> Terrakion</td> 261</td></tr>
640</td> Virizion</td> 261</td></tr>
641</td> Tornadus</td> 261</td></tr>
642</td> Thundurus</td> 261</td></tr>
643</td> Reshiram</td> 306</td></tr>
644</td> Zekrom</td> 306</td></tr>
645</td> Landorus</td> 270</td></tr>
646</td> Kyurem</td> 297</td></tr>
647</td> Keldeo</td> 261</td></tr>
648</td> Meloetta</td> 270</td></tr>
649</td> Genesect</td> 270</td></tr>
716</td> Xerneas</td> 306</td></tr>
717</td> Yveltal</td> 306</td></tr>
718</td> Zygarde</td> 270</td></tr>
719</td> Diancie</td> 270</td></tr>
720</td> Hoopa</td> 270</td></tr>
721</td> Volcanion</td> 270</td></tr>
785</td> Tapu Koko</td> 257</td></tr>
786</td> Tapu Lele</td> 257</td></tr>
787</td> Tapu Bulu</td> 257</td></tr>
788</td> Tapu Fini</td> 257</td></tr>
793</td> Nihilego</td> 257</td></tr>
794</td> Buzzwole</td> 257</td></tr>
795</td> Pheromosa</td> 257</td></tr>
796</td> Xurkitree</td> 257</td></tr>
797</td> Celesteela</td> 257</td></tr>
798</td> Kartana</td> 257</td></tr>
799</td> Guzzlord</td> 257</td></tr>
800</td> Necrozma</td> 270</td></tr>
801</td> Magearna</td> 270</td></tr>
802</td> Marshadow</td> 270</td></tr>
805</td> Stakataka</td> 257</td></tr>
806</td> Blacephalon</td> 257</td></tr>
807</td> Zeraora</td> 270</td></tr>
</tbody></table>
So these Pokemon are nearly all ‘Legendary’. They are big end-game
Pokemon, with real rarity in game. They also don’t evolve from, or to,
anything, so our rule doesn’t classify them effectively.</p>
pokemon_evolves_from %>%</span> 
  filter</span></span>(</span>base_experience <</span> 200</span> &</span> base_experience ></span> 100</span> &</span> power_level ==</span> 1</span>)</span></span> %>%</span> 
  select</span></span>(</span>id,</span> name,</span> base_experience)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
id</th> name</th> base_experience</th></tr></thead>

83</td> Farfetch’d</td> 132</td></tr>
115</td> Kangaskhan</td> 172</td></tr>
127</td> Pinsir</td> 175</td></tr>
128</td> Tauros</td> 172</td></tr>
131</td> Lapras</td> 187</td></tr>
132</td> Ditto</td> 101</td></tr>
142</td> Aerodactyl</td> 180</td></tr>
201</td> Unown</td> 118</td></tr>
203</td> Girafarig</td> 159</td></tr>
206</td> Dunsparce</td> 145</td></tr>
213</td> Shuckle</td> 177</td></tr>
214</td> Heracross</td> 175</td></tr>
222</td> Corsola</td> 144</td></tr>
225</td> Delibird</td> 116</td></tr>
227</td> Skarmory</td> 163</td></tr>
234</td> Stantler</td> 163</td></tr>
241</td> Miltank</td> 172</td></tr>
302</td> Sableye</td> 133</td></tr>
303</td> Mawile</td> 133</td></tr>
311</td> Plusle</td> 142</td></tr>
312</td> Minun</td> 142</td></tr>
313</td> Volbeat</td> 151</td></tr>
314</td> Illumise</td> 151</td></tr>
324</td> Torkoal</td> 165</td></tr>
327</td> Spinda</td> 126</td></tr>
335</td> Zangoose</td> 160</td></tr>
336</td> Seviper</td> 160</td></tr>
337</td> Lunatone</td> 161</td></tr>
338</td> Solrock</td> 161</td></tr>
351</td> Castform</td> 147</td></tr>
352</td> Kecleon</td> 154</td></tr>
357</td> Tropius</td> 161</td></tr>
359</td> Absol</td> 163</td></tr>
369</td> Relicanth</td> 170</td></tr>
370</td> Luvdisc</td> 116</td></tr>
417</td> Pachirisu</td> 142</td></tr>
440</td> Happiny</td> 110</td></tr>
441</td> Chatot</td> 144</td></tr>
442</td> Spiritomb</td> 170</td></tr>
455</td> Carnivine</td> 159</td></tr>
479</td> Rotom</td> 154</td></tr>
538</td> Throh</td> 163</td></tr>
539</td> Sawk</td> 163</td></tr>
550</td> Basculin</td> 161</td></tr>
556</td> Maractus</td> 161</td></tr>
561</td> Sigilyph</td> 172</td></tr>
587</td> Emolga</td> 150</td></tr>
594</td> Alomomola</td> 165</td></tr>
615</td> Cryogonal</td> 180</td></tr>
618</td> Stunfisk</td> 165</td></tr>
621</td> Druddigon</td> 170</td></tr>
626</td> Bouffalant</td> 172</td></tr>
631</td> Heatmor</td> 169</td></tr>
632</td> Durant</td> 169</td></tr>
676</td> Furfrou</td> 165</td></tr>
701</td> Hawlucha</td> 175</td></tr>
702</td> Dedenne</td> 151</td></tr>
707</td> Klefki</td> 165</td></tr>
741</td> Oricorio</td> 167</td></tr>
764</td> Comfey</td> 170</td></tr>
765</td> Oranguru</td> 172</td></tr>
766</td> Passimian</td> 172</td></tr>
771</td> Pyukumuku</td> 144</td></tr>
772</td> Type: Null</td> 107</td></tr>
774</td> Minior</td> 154</td></tr>
775</td> Komala</td> 168</td></tr>
776</td> Turtonator</td> 170</td></tr>
777</td> Togedemaru</td> 152</td></tr>
778</td> Mimikyu</td> 167</td></tr>
779</td> Bruxish</td> 166</td></tr>
780</td> Drampa</td> 170</td></tr>
781</td> Dhelmise</td> 181</td></tr>
803</td> Poipole</td> 189</td></tr>
</tbody></table>
These Pokemon are not end game, but they either have very short
evolution trees (2 Pokemon long), or no evolution tree at all.</p>
So our ‘group count’ process doesn’t work well, and neither does our
‘evolves_from_species</code>’ process. We’re going to have to to learn some
new moves.</p>
TM01 (e.g. Tidy Models 01)</h2>
Clustering is the process of using machine learning to derive a
categorical variable from data. The simplest form of clustering that
seems relevant to our problem is k-means. Seeing as we have a pretty
good intuition that 3 groups implicitly</em> exist in this data, and a
clear visualisation</em> supporting us, lets cut straight to asking R for 3
groups. k-means clustering aims to divide the data into a known</em> number
of groups which is set in the centers</code> argument, and doesn’t require
any data to be fed to it as examples of what makes up a ‘group’. That
sentence might be a little confusing, as we do obviously give it some
data. What we don’t</em> give it is a training data set which has examples
of what Pokemon are supposed to be in a given group, labelled with the
group they are supposed to be in, e.g. Squirtle is in group 1, Wartortle
is in group 2, Blastoise is in group 3, and then give it unlabelled data
to classify, e.g. “What group is Charmeleon in?”. k-means will figure
out</em> how to split the continuous variable base_experience</code> into 3
groups.</p>
set.seed</span></span>(</span>68</span>)</span></span>

pokemon %>%</span> 
  select</span></span>(</span>base_experience)</span></span>  %>%</span> 
  kmeans</span></span>(</span>centers</span> =</span> 3</span>)</span></span> %>%</span> 
  augment</span></span>(</span>pokemon)</span></span> -></span> pokemon_cluster
</span></code></pre>
How kmeans works</h3>
First, we set centroids to be at random positions in the data. To make
sure this doesn’t effect consistency in my article I’ve used set.seed</code>
so k-means starts looking for the centres of our clusters from the same
position each time. A ‘centroid’ can be seen as a ‘centre point’ for
each cluster. We have the same number of centroids set as the value set
in the centers</code> argument. Each data point is then assigned to it’s
closest centroid.</p>

The Sum of Squared Errors (SSE) from the centroids is then used as an
objective function</em> towards a local minimum</em>.</p>
</blockquote>
This is the core concept of how k-means calculates a solution. The Sum
of Squared Errors are calculated like this:</p>
{% katex %}
\sum^n_{i-1}(x_i-\bar{x})^2
{% endkatex %}</p>
What this means is for each group, the distance from the centroid to
each observation is measured, and then squared. Then each of those
squared distances, one per observation in the group, is totalled.</p>
The centroid is then moved to the average</em> value of it’s group. Because
the centroids are now no longer in the same position as when the SSE was
calculated, the SSE is now recalculated for all observations, to each of
the new</em> centroid positions. This means that some observations are now
closer to a different</em> centroid, and so get assigned to a different</em>
cluster.</p>
Then the centroids are moved again</em> to the new</em> average of the new</em>
cluster. Each cluster then gets new distances</em> calculated. This will go
on until termination when, each centroid is in the average position of
the cluster, and each observation in the cluster is closest to the
centroid</em> it is currently assigned to.</p>
That’s a slightly wordy description of a complicated process. I
recommend that you have a look at the k-means explanation in tidy
models</a> to really
cement the concept. It also contains the most adorable animation of a
statistical concept in existance</em>.</p>
How to use it</h3>
Here, I’ve simply selected the one column of data, base_experience</code>,
and piped it into kmeans</code>, which is part of base R. This returns a
super un-tidy list like object of class "kmeans"</code>, however, with
tidymodels</code> we can easily make it usable.</p>
augment</code> helps us to match the output of the kmeans</code> function back to
our data for easier processing. The output of kmeans</code> doesn’t actually
contain any of the other information about our data, it only got given
one column remember? augment</code> goes through the return of kmeans</code>,
finds the relevant bit, and matches it neatly back to our original data
ready for plotting in one step!</p>
pokemon_cluster %>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> id,</span> y</span> =</span> base_experience,</span> colour</span> =</span> .cluster)</span></span>)</span></span> +</span>
  geom_point</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Power level by kmeans clustering"</span></span>)</span></span>
</span></code></pre>
</p>
This is a lot</em> better. It gives us really clear groups, in exactly
where we expected them. It’s also tonnes simpler code!</p>
Lets check some of our boundary positions, just to make sure it makes
sense.</p>
pokemon_cluster %>%</span> 
  filter</span></span>(</span>.cluster ==</span> 1</span> &</span> base_experience ></span> 100</span>)</span></span> %>%</span> 
  select</span></span>(</span>id,</span> name,</span> base_experience)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
id</th> name</th> base_experience</th></tr></thead>

132</td> Ditto</td> 101</td></tr>
440</td> Happiny</td> 110</td></tr>
699</td> Aurorus</td> 104</td></tr>
762</td> Steenee</td> 102</td></tr>
772</td> Type: Null</td> 107</td></tr>
</tbody></table>
pokemon_cluster %>%</span> 
  filter</span></span>(</span>.cluster ==</span> 2</span> &</span> base_experience ></span> 200</span>)</span></span> %>%</span> 
  select</span></span>(</span>id,</span> name,</span> base_experience)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
id</th> name</th> base_experience</th></tr></thead>

189</td> Jumpluff</td> 207</td></tr>
</tbody></table>
Ditto and Type: Null are just a plain weird Pokemon due to their
abilities messing with their type. Happiny is a baby type, but from a
family with an insane</em> base_experience. Jumpluff is an awkward edge
case. Technically the 3rd evolution, it’s still got an extremely</em> low
base_experience. Potentially this is for game balancing as they are
relatively regularly encountered? Aurorus and Steenee I do not have a
good hypothesis for.</p>
Generally I think that this is a pretty good solution. There are maybe a
few edge cases that are open to interpretation, but that’s just what we
get sometimes with machine learning. Lacking a labelled training data
set, we can’t compute a confusion matrix or ROC-curve.</p>
Conclusion</h2>
We’ve found a situation in the ‘real’ world where we know from context
there is a categorical relationship, but from the data available it’s
not possible to classify that precisely. However we can create this
categorisation using machine learning! Even better, we can use the
tidymodels</code> package to help us do it quickly and cleanly.</p>


The Missingno Experiment and Multiple Form Pokemon
2020-07-01T00:00:00+00:00
Wild missingno appeared!</h2>

Missingno is the patron Pokemon of data science. You’re just casually
surfing up and down your data, doing some sweet coding, when suddenly a
bunch of missing and corrupted data gets in you way, and you suddenly
have a bunch of random items in your bag for no reason. OK, well maybe I
just have a messy bag.</p>
The valuable part of this metaphor is the part where you battle
Missingno, and win. I’ve been doing this with my Pokedex project
recently, to try and iron out what data I can rely on from my data
source, and what’s a bit patchy.</p>
library</span></span>(</span>pokedex)</span></span>
library</span></span>(</span>tidyverse)</span></span>
library</span></span>(</span>naniar)</span></span>
library</span></span>(</span>skimr)</span></span>
</span></code></pre>
Go, Skimr!</h2>
Skimr gives us a text based summary view. As well as the basics on data
set size, it also shows us some statistical values, but most valuably it
describes how many values are missing, and in what columns.</p>
pokemon %>%</span> 
  skimr::</span>skim</span></span>(</span>)</span></span>
</span></code></pre>
</th> </th></tr></thead>

Name</td> Piped data</td></tr>
Number of rows</td> 807</td></tr>
Number of columns</td> 24</td></tr>
_______________________</td> </td></tr>
Column type frequency:</td> </td></tr>
character</td> 8</td></tr>
list</td> 1</td></tr>
numeric</td> 15</td></tr>
________________________</td> </td></tr>
Group variables</td> None</td></tr>
</tbody></table>
Data summary</p>
Variable type: character</strong></p>
skim_variable</th> n_missing</th> complete_rate</th> min</th> max</th> empty</th> n_unique</th> whitespace</th></tr></thead>

identifier</td> 0</td> 1.00</td> 3</td> 20</td> 0</td> 807</td> 0</td></tr>
type_1</td> 0</td> 1.00</td> 3</td> 8</td> 0</td> 18</td> 0</td></tr>
type_2</td> 402</td> 0.50</td> 3</td> 8</td> 0</td> 18</td> 0</td></tr>
name</td> 0</td> 1.00</td> 3</td> 12</td> 0</td> 807</td> 0</td></tr>
genus</td> 0</td> 1.00</td> 11</td> 21</td> 0</td> 589</td> 0</td></tr>
color</td> 20</td> 0.98</td> 3</td> 6</td> 0</td> 10</td> 0</td></tr>
shape</td> 20</td> 0.98</td> 4</td> 9</td> 0</td> 14</td> 0</td></tr>
habitat</td> 422</td> 0.48</td> 3</td> 13</td> 0</td> 9</td> 0</td></tr>
</tbody></table>
Variable type: list</strong></p>
skim_variable</th> n_missing</th> complete_rate</th> n_unique</th> min_length</th> max_length</th></tr></thead>

flavour_text</td> 0</td> 1</td> 807</td> 1</td> 1</td></tr>
</tbody></table>
Variable type: numeric</strong></p>
skim_variable</th> n_missing</th> complete_rate</th> mean</th> sd</th> p0</th> p25</th> p50</th> p75</th> p100</th> hist</th></tr></thead>

id</td> 0</td> 1.00</td> 404.00</td> 233.11</td> 1.0</td> 202.5</td> 404</td> 605.5</td> 807.0</td> ▇▇▇▇▇</td></tr>
species_id</td> 0</td> 1.00</td> 404.00</td> 233.11</td> 1.0</td> 202.5</td> 404</td> 605.5</td> 807.0</td> ▇▇▇▇▇</td></tr>
height</td> 0</td> 1.00</td> 1.16</td> 1.08</td> 0.1</td> 0.6</td> 1</td> 1.5</td> 14.5</td> ▇▁▁▁▁</td></tr>
weight</td> 0</td> 1.00</td> 61.77</td> 111.52</td> 0.1</td> 9.0</td> 27</td> 63.0</td> 999.9</td> ▇▁▁▁▁</td></tr>
base_experience</td> 0</td> 1.00</td> 144.85</td> 74.95</td> 36.0</td> 66.0</td> 151</td> 179.5</td> 608.0</td> ▇▇▁▁▁</td></tr>
is_default</td> 0</td> 1.00</td> 1.00</td> 0.00</td> 1.0</td> 1.0</td> 1</td> 1.0</td> 1.0</td> ▁▁▇▁▁</td></tr>
hp</td> 0</td> 1.00</td> 68.75</td> 26.03</td> 1.0</td> 50.0</td> 65</td> 80.0</td> 255.0</td> ▃▇▁▁▁</td></tr>
attack</td> 0</td> 1.00</td> 76.09</td> 29.54</td> 5.0</td> 55.0</td> 75</td> 95.0</td> 181.0</td> ▂▇▆▂▁</td></tr>
defense</td> 0</td> 1.00</td> 71.73</td> 29.73</td> 5.0</td> 50.0</td> 67</td> 89.0</td> 230.0</td> ▅▇▂▁▁</td></tr>
special_attack</td> 0</td> 1.00</td> 69.49</td> 29.44</td> 10.0</td> 45.0</td> 65</td> 90.0</td> 173.0</td> ▃▇▅▂▁</td></tr>
special_defense</td> 0</td> 1.00</td> 70.01</td> 27.29</td> 20.0</td> 50.0</td> 65</td> 85.0</td> 230.0</td> ▇▇▂▁▁</td></tr>
speed</td> 0</td> 1.00</td> 65.83</td> 27.74</td> 5.0</td> 45.0</td> 65</td> 85.0</td> 160.0</td> ▃▇▆▂▁</td></tr>
generation_id</td> 20</td> 0.98</td> 3.67</td> 1.94</td> 1.0</td> 2.0</td> 4</td> 5.0</td> 7.0</td> ▇▅▃▅▅</td></tr>
evolves_from_species_id</td> 426</td> 0.47</td> 364.35</td> 232.43</td> 1.0</td> 156.0</td> 345</td> 570.0</td> 803.0</td> ▇▆▅▆▅</td></tr>
evolution_chain_id</td> 20</td> 0.98</td> 195.96</td> 124.57</td> 1.0</td> 84.0</td> 187</td> 303.0</td> 427.0</td> ▇▆▅▆▅</td></tr>
</tbody></table>
I was expecting some missing data in type_2</code>, and
evolves_from_species_id</code>, but I wasn’t expecting only half of habitat</code>
to be there. Either I broke something in my data pipeline, or the data
wasn’t there to begin with. colour</code>, shape</code>, generation_id</code> and
evolution_chain_id</code> are all missing 20 entries each, which is a bit or
a coincidence. I wonder if they are all missing from the same Pokemon?</p>
Visdat I choose you!</h2>
visdat</code> is a package that helps you visualise missing data and data
types.</p>
visdat::</span>vis_dat</span></span>(</span>pokemon)</span></span>
</span></code></pre>
</p>
This clearly shows us the data types in each column, and where values
are missing in context. It looks like habitat might just not be
available after a certain time. It also looks like colour</code>, shape</code>,
generation_id</code> and evolution_chain_id</code> looks like they are maybe all
missing from the same individual Pokemon?</p>
Go, Naniar!</h2>
Naniar</code> helps us check through plots where relationships between
missing values and other variables might occur. Lets check first if
there is a relationship between generation_id</code> and evolution_chain_id</code></p>
pokemon %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>generation_id,</span> evolution_chain_id)</span></span>)</span></span> +</span>
  geom_miss_point</span></span>(</span>)</span></span>
</span></code></pre>
</p>
This plot might need a little explanation. For the Not Missing</code> blue
values, this is a normal geom_point()</code>. However, where the values are
marked as Missing</code> pink they are deliberately moved below the (0,0)</code>
mark for the axis they are missing values for</em>, then they ‘jitter’, to
avoid over-plotting. The little cluster at the far bottom left in a line
marks that for all</em> values where evolution_chain_id</code> being missing,
generation_id</code> is also missing. Let’s have a look at the
evolves_from_species_id</code> variable just to help us understand.</p>
pokemon %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>evolves_from_species_id,</span> generation_id)</span></span>)</span></span> +</span>
  geom_miss_point</span></span>(</span>)</span></span>
</span></code></pre>
</p>
This is showing that in every game generation (Red/Blue, X/Y, etc.) that
there are Pokemon that have an evolves_from_species_id</code>, i.e. they have
a precursor Pokemon, and that there are also Pokemon that don’t</em> have a
precursor. Just what we see in the games. It’s also showing that have
neither generation_id</code> or evolves_from_species_id</code>.</p>
Who is that Pokemon?</h2>
Now we know the characteristics of the missing data we are interested
in, we can pull them out easily. Especially with the newly released
across()</code> function</a></p>
missing_cols \u003c-</span> c</span></span>(</span>"</span>color"</span></span>,</span> "</span>shape"</span></span>,</span> "</span>generation_id"</span></span>,</span> "</span>evolves_from_species_id"</span></span>)</span></span>
pokemon %>%</span> 
  filter</span></span>(</span>across</span></span>(</span>missing_cols,</span> ~</span>is.na</span></span>(</span>.x)</span></span>)</span></span>)</span></span> %>%</span> 
  select</span></span>(</span>name,</span> identifier,</span> missing_cols)</span></span> -></span> missing_pokes

missing_pokes %>%</span> knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
name</th> identifier</th> color</th> shape</th> generation_id</th> evolves_from_species_id</th></tr></thead>

Deoxys</td> deoxys-normal</td> NA</td> NA</td> NA</td> NA</td></tr>
Wormadam</td> wormadam-plant</td> NA</td> NA</td> NA</td> NA</td></tr>
Giratina</td> giratina-altered</td> NA</td> NA</td> NA</td> NA</td></tr>
Shaymin</td> shaymin-land</td> NA</td> NA</td> NA</td> NA</td></tr>
Basculin</td> basculin-red-striped</td> NA</td> NA</td> NA</td> NA</td></tr>
Darmanitan</td> darmanitan-standard</td> NA</td> NA</td> NA</td> NA</td></tr>
Tornadus</td> tornadus-incarnate</td> NA</td> NA</td> NA</td> NA</td></tr>
Thundurus</td> thundurus-incarnate</td> NA</td> NA</td> NA</td> NA</td></tr>
Landorus</td> landorus-incarnate</td> NA</td> NA</td> NA</td> NA</td></tr>
Keldeo</td> keldeo-ordinary</td> NA</td> NA</td> NA</td> NA</td></tr>
Meloetta</td> meloetta-aria</td> NA</td> NA</td> NA</td> NA</td></tr>
Meowstic</td> meowstic-male</td> NA</td> NA</td> NA</td> NA</td></tr>
Aegislash</td> aegislash-shield</td> NA</td> NA</td> NA</td> NA</td></tr>
Pumpkaboo</td> pumpkaboo-average</td> NA</td> NA</td> NA</td> NA</td></tr>
Gourgeist</td> gourgeist-average</td> NA</td> NA</td> NA</td> NA</td></tr>
Oricorio</td> oricorio-baile</td> NA</td> NA</td> NA</td> NA</td></tr>
Lycanroc</td> lycanroc-midday</td> NA</td> NA</td> NA</td> NA</td></tr>
Wishiwashi</td> wishiwashi-solo</td> NA</td> NA</td> NA</td> NA</td></tr>
Minior</td> minior-red-meteor</td> NA</td> NA</td> NA</td> NA</td></tr>
Mimikyu</td> mimikyu-disguised</td> NA</td> NA</td> NA</td> NA</td></tr>
</tbody></table>
So it looks like in the current version of the package, these Pokemon
all have ‘complex’ identifiers. This is because these Pokemon all have
different forms. Some vary by colour like
Basculin</a>
which can be Red or Blue striped, others have ability transformations,
like
Aegislash</a>
or which game it was caught in like
Deoxys</a>.</p>
missing_pokes %>%</span> 
  pull</span></span>(</span>name)</span></span> %>%</span> 
  stringr::</span>str_to_lower</span></span>(</span>.)</span></span> -></span> missing_pokes_name_list

pokedex$</span>pokemon_species %>%</span>
  filter</span></span>(</span>
    stringr::</span>str_to_lower</span></span>(</span>identifier)</span></span> %in%</span> missing_pokes_name_list
    )</span></span> %>%</span> 
  select</span></span>(</span>identifier,</span> generation_id,</span> evolves_from_species_id,</span> shape_id,</span> color_id)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
identifier</th> generation_id</th> evolves_from_species_id</th> shape_id</th> color_id</th></tr></thead>

deoxys</td> 3</td> NA</td> 12</td> 8</td></tr>
wormadam</td> 4</td> 412</td> 5</td> 5</td></tr>
giratina</td> 4</td> NA</td> 10</td> 1</td></tr>
shaymin</td> 4</td> NA</td> 8</td> 5</td></tr>
basculin</td> 5</td> NA</td> 3</td> 5</td></tr>
darmanitan</td> 5</td> 554</td> 8</td> 8</td></tr>
tornadus</td> 5</td> NA</td> 4</td> 5</td></tr>
thundurus</td> 5</td> NA</td> 4</td> 2</td></tr>
landorus</td> 5</td> NA</td> 4</td> 3</td></tr>
keldeo</td> 5</td> NA</td> 8</td> 10</td></tr>
meloetta</td> 5</td> NA</td> 12</td> 9</td></tr>
meowstic</td> 6</td> 677</td> 6</td> 2</td></tr>
aegislash</td> 6</td> 680</td> 5</td> 3</td></tr>
pumpkaboo</td> 6</td> NA</td> 1</td> 3</td></tr>
gourgeist</td> 6</td> 710</td> 5</td> 3</td></tr>
oricorio</td> 7</td> NA</td> 9</td> 8</td></tr>
lycanroc</td> 7</td> 744</td> 8</td> 3</td></tr>
wishiwashi</td> 7</td> NA</td> 3</td> 2</td></tr>
minior</td> 7</td> NA</td> 1</td> 3</td></tr>
mimikyu</td> 7</td> NA</td> 2</td> 10</td></tr>
</tbody></table>
If we go back to the raw source data, we can see that the data is
actually there for most cases, it just didn’t join properly because in
the source data, they are identified by the simple name, in lower case,
and in this version of the
package</a>
this data is joined on id</code> AND the column that actually has the complex
name. Also, because shape and color link through</em> this data, they are
missed as well!</p>
You defeated wild missingno!</h2>
This is all based on my Pokedex R data package, which I’m just about to
fix :)</p>
daveparr/pokedex</a></p>


Why did I make this dev.to API wrapper?
2020-06-30T00:00:00+00:00
dev.to.ol</code> is 0.0.1!</p>
daveparr/dev.to.ol</a></p>
What’s in the box?</h2>
dev.to.ol has the minimum set of viable functions I think it needs to
operate, and (just) enough testing to keep it stable. The api is
starting to come together and there seems to be at least 1 other person
who cares enough about this project to talk about it to someone else, so
I thought it might be handy to get a little more together about things.
I also wanted to make up a celebration to share a little about why I
made it and what dev.to means to me.</p>
Are there other similar boxes?</h2>
R is in many ways a literate programming language. Markdown is pretty
baked into most of our ecosystem through R
Markdown</a>, as is
LaTeX</a>. I think because of this,
and the frequent usage in academic publishing, R users have developed
tooling for blogging pretty heavily. We have
blogdown</a> and
distill</a> which are becoming very
widely used and feature rich, and @maelle has written a pretty
comprehensive guide to all the different approaches in the Scientific
Blogging with R Markdown
course</a> including her own
solution for Wordpress,
goodpress</a>. As Data Scientists
(emphasis on the science</em>), we need to share our work reproducibly, and
in a way that encourages understanding and review.</p>
So why have another box?</h2>
I was looking for something to keep me sane while I was furloughed.</p>
Also, I had poked around dev.to about a year before, and was impressed
by what I took to be it’s un-official mission statement of “We aren’t
Medium, we’re what Medium should have been”.</p>
Medium was never meant to be a part of the developer ecosystem</a></p>
I’m sure there is a lot more to it than that, especially with the recent
Forem</a>
announcement, but personally, that was a hook I perceived which really
stuck me.</p>
However, at the time the in built editor was “fine”. Fine enough, but
not great. I’m also used to R Notebooks, where I write my code next to
my prose, and compile the whole thing in one. It seemed valuable to
bring that ability to my dev.to posts. I also was aware of all the
alternatives above, but dev.to offered 2 things that were different to
the above options:</p>
Hosted and managed service</h3>
Don’t get me wrong, I have nothing but love for JAMstack and static
sites in general. I maintain a template for making them for SatRdays
conferences</a>.
However, I don’t love the process of doing it. In this case I absolutely
believe in the cause, which is why I continue to volunteer on the
project. However, the process of interacting with static sites doesn’t
actually make me happy</em>.</p>
I was looking for the simplest way I could casually blog, which ideally
required no extra work beyond writing articles (to start with).</p>
Community</h3>
I wanted at least a few people to read my articles, and I wanted to read
theirs. I’m sure I could have accomplished this with more work in SEO,
cultivating a social media network and all that jazz, but again: I just
wanted to write some stuff that some people might read, and read their
stuff. dev.to had a discovery feed, it had active users, it had
community. This was just what I was looking for. I also kind of hate
twitter (yes, I still use it when I ‘have’ to).</p>
Motivation</h2>
Initially my work flow sucked. I wrote an .Rmd</code> in RStudio. Compiled it
to a GitHub flavoured .md</code>. Copy and pasted the output into the editor,
added the meta data, then uploaded all the images. If I found I’d made a
mistake, I’d either do the right thing, which was edit the .Rmd</code>,
recompile, re-copy-pasta, or I’d more often do the quick thing which is
enter the browser editor and fix the typo.</p>
Inception</h2>
As I was chewing over the best way to make this work gooder (I am a
programmer after all), a few things happened simultaneously.</p>
Most of the R users on the site syndicate</h3>
There are some really great R users on here already:</p>
juliasilge</a></p>
sckott</a></p>
maelle</a></p>
colinfay</a></p>
I found out that most of the R users here I follow are actually
re-syndicating through rss</a>.</p>
That’s great for them, but I was actually trying to avoid my own site!</p>
Dev.to has an API</h3>
I was poking around older posts and found out that dev.to has an
API</a>. At the time I thought I might play with
webhooks to boot ‘saved’ articles into pocket to work on my e-reader (at
some point I still might). However, in this case it was kind of perfect.
An API driven work flow between .Rmd in RStudio and the hosted dev.to
community. I could see it in my head, and it’s not like I was busy in
May…</p>
posting from .Rmd to dev.to</a></p>
Evolution</h2>
After proving it ‘might’ be doable, I did some work fleshing out the
‘best’ way to do it.</p>
posting straight from .Rmd to dev.to (for real this time)</a></p>
So I started some super-basic minimum viable functions. How do I get the
post to turn up on dev.to? How do I make the meta-data work with the
post? What happens if I’ve published and need to correct a typo? All
solvable, all import features, and all now implemented and tested. I got
to learn a lot about testing API wrappers with vcr</code> thanks to @sckott.</p>
Testing my dev.to.ol package</a></p>
The functions in the package changed a bit as I went through user
testing by using it to write my posts on dev.to that you can see here. I
was also able to generate content pretty quick because I could write
about developing the functions that I was testing at the same time by
writing the articles. Virtuous cycles! (/ black holes)</p>
</p>
At this point I was also starting to look for new work, and was
wondering about if I had made the choice backwards. Maybe I did need my
own site after all. Somewhere to put my CV, and that looked a bit more
professional, and maybe didn’t have so much clutter of articles mixed
with my own questions and a bunch of content from other people. Then I
discovered the @stackbit integration!</p>
You can now generate self-hostable static blogs right from your DEV content via Stackbit</a></p>
This meant that I could hand off all the hosting and styling and
management and deploys to them, but still get a hugo repo which I can
tailor how I want, such as removing the #meta</code> and #help</code> posts, which
wouldn’t be useful to a recruiter, but also automatically put my content
into a presentable site, with a few contextual links to thing like my
GitHub and LinkedIn under my personal daveparr.info</a>
domain.</p>
I made my dev.to content into a blog to find a new job</a></p>
Future</h2>
The primary future goal is finding a good way to reference images in the
article in a way that dev.to can use. It’s likely that this will end up
being github itself. Additionally better testing is probably on the
cards as I go. The functions and package API have stabilised enough now
that this can be comprehensive, and getting some CI/CD would be nice
too. I’m also planning on working on smarter ways to run analytics on
the data you can get back from the API about your posts. Maybe even an
inbuilt shiny app?</p>
I hope some others of you might find the package useful, and maybe this
might motivate you to share the work you might already be doing in R, or
even pick up R as a new language!</p>
Photo by Erwan Hesry</a> via
Unsplash</p>


Nativefier is bonkers
2020-06-26T00:00:00+00:00
I made an electron app in 4 lines…</p>
nativefier</span></span> "</span>http://musicforprogramming.net/"</span></span> -</span>n</span> "</span>musicforprogramming"</span></span></span>
cd</span></span> musicforprogramming-linux-x64/</span>
sudo</span></span> chmod +x musicforprogramming</span>
./musicforprogramming</span></span>
</span></code></pre>
Motivation</h1>
I like music, but don’t like music in browser tabs. Basically because i have it open all the time, I want to find and control it easily, and I don’t want it cluttering up an area that might be soley focused on work. </p>
I discovered some neat apps for google play, and wanted to see if there was one for my other go to, musicforprogramming</a>. There wasn’t, so I just casually googled how to convert a page into an electron app and OMFG!</p>
Solution</h1>
Nativefier</a></p>
This post</a> and this one</a> helped iron out some kinks and now I can launch programming music right from my VS code terminal!</p>
</p>
Extra credit</h1>
alias</span> musicforprogramming</span>=</span>"</span>~/Dev/musicforprogramming-linux-x64/musicforprogramming
</span></span></span></span></code></pre>


Building my first Django project with CSS and Static Files
2020-06-25T00:00:00+00:00
I’m working through Django for Beginners by William S. Vincent</a>. Until now we’ve had some pretty barebones Times New Roman style UI, however that’s all about to change! I think.</p>
I’m really appreciating this incremental iteration of concepts in each new Chapter. Each time I start a new app for the chapter it’s engraining the muscle memory and concept recall into me. I’m able to run through the first 5 lines of code at the CLI nearly from memory. It’s also a great mechanism to help you if you get stuck on a wierd error and don’t know what to do. You know the next chapter will have a clean slate. Great idea William.</p>
The repetition isn’t just the set-up though.</p>

“Now we can add the functionality for individual blog pages. How do we do that? We need to create a new view, url, and template. I hope you’re noticing a pattern in development with Django now!”</p>
</blockquote>
I sure am. </p>
I’m glad that the book also doesn’t go into non-django areas, but still points readers to places to look for more info. The last chapter pointed me towards some resources to understand more about databases, and this one does similar with CSS.</p>
It was interesting to see that the approach used to identify a blog post for navigation in the URL patterns looked kinda</em> regex-ish:</p>
urlpatterns</span></span> =</span> [</span>
    path</span></span></span></span>(</span>'</span></span></span>post/<int:pk>/'</span></span></span>,</span> BlogDetailView</span></span>.</span></span>as_view</span></span></span></span>(</span>)</span></span>,</span> name</span>=</span>'</span></span></span>post_detail'</span></span></span>)</span></span>,</span>
    path</span></span></span></span>(</span>'</span></span></span>'</span></span></span>,</span> BlogListView</span></span>.</span></span>as_view</span></span></span></span>(</span>)</span></span>,</span> name</span>=</span>'</span></span></span>home'</span></span></span>)</span></span>,</span>
]</span></span>
</span></code></pre>
<int:pk></code> is the part that identifies the blog post required for the url to link from the ListView to the DetailView. It was interesting to find out there was something like a unique identifier for each post baked into it in the background. I was wondering about how this primary key is treated in bigger projects. Is it common to keep this approach of integer ordered primary keys, or do they get replaced in larger, more complex projects with hashes or other unique ids?</p>


Why use AWS Lambda for Data Science?
2020-06-23T00:00:00+00:00
Motivation</h2>
Serverless is a way to deploy code, without having to manage the
infrastructure underneath it. In AWS terms this means there is an
compute instance that runs your code, except you don’t control it, and
that might be a good thing. It only exists when it’s asked for by
something else, and therefore you only pay for the work it does. If you
need to do work concurrently you get a new instance, which also goes
away as soon as you don’t need it, so it’s scaleable.</p>
For a data scientist this is an interesting prospect for a number of
reasons. The first is keeping you hands clean. Not all people in this
role come from a ‘operations’ background. Many of us are analysts first,
and graduate into the role. However, that shouldn’t mean we don’t ‘own
our deployments’. However, it also means that we might not have the
background, time or inclination to really get into the nitty-gritty.
Managed infrastructure, that can scale seamlessly out of the box is a
nice middle ground. We can still manage our own deployments, but theres
less to worry about than owning your own EC2 instances, let alone a
fleet of them. The way that the instances themselves die off is also
valuable. We may be doing work that requires 24/7 processing, but often,
we aren’t. Why pay for a box which might have 50% required utilisation
time, or even less?</p>
Limits</h3>
Just like in everything there is a balance. There are physical</em> limits
to this process. I’ve had success deploying data science assets in this
architecture, but if you can’t fit your job in these limits, this
already isn’t for you. Sure, data science can be</em> giant machine
learning models on huge hardware with massive data volumes, but we have
to be honest and acknowledge that it isn’t always. K.I.S.S. should apply
to everything.</p>
If you can get good enough business results with a linear regression,
don’t put in 99% more effort to train the new neural network hotness to
get a 2% increase in performance. Simplicity in calculation, deployment,
and explainability matter</em>.</p>

“No ML is easier to manage than no ML” ©
@julsimon</a></p>
</blockquote>
Getting started</h2>
from</span></span> scipy</span></span> import</span></span></span></span></span> stats</span></span>
import</span> numpy</span></span> as</span> np</span></span></span>

np</span>.</span>random</span></span>.</span></span>seed</span></span></span></span>(</span>12345678</span>)</span></span>

x</span></span> =</span> np</span>.</span>random</span></span>.</span></span>random</span></span></span></span>(</span>10</span>)</span></span>
y</span></span> =</span> 1.</span>6</span>*</span>x</span></span> +</span> np</span>.</span>random</span></span>.</span></span>random</span></span></span></span>(</span>10</span>)</span></span>

slope</span></span>, intercept</span></span>, r_value</span></span>, p_value</span></span>, std_err</span></span> =</span> 
  stats</span></span>.</span></span>linregress</span></span></span></span>(</span>x</span></span>,</span> y</span></span>)</span></span>
</span></code></pre>
This is a nonsense linear regression. IMHO it’s a data science ‘Hello
World’. Let’s make it an AWS Lambda serverless function.</p>
+</span> import json
</span>from scipy import stats
import numpy as np
+</span> def lambda_handler(event, context):
</span>  np.random.seed(12345678)

x = np.random.random(10)
y = 1.6*x + np.random.random(10)

slope, intercept, r_value, p_value, std_err = stats.linregress(x, y) 
+</span>   return_body = {
</span>  +       "m": slope, "c": intercept,"r2": r_value ** 2, 
  +       "p": p_value, "se": std_err
  +   }
+</span>   return {"body": json.dumps(return_body)}
</span></span></code></pre>
These changes achieve 3 things:</p>

Turning a script</em> into a function</em></li>
Supplying the function arguments event</code> and context</code></li>
Formatting the return as json</li>
</ol>
These are required as AWS Lambda needs a function</em>. This is so that its
event driven architecture</em> can feed in data through event</code>, and so
that it’s json</code> formatted data can both be received by your function,
and then also the response be returned by that function into the rest of
the system.</p>
You can then open up the AWS console in a browser, navigate to the
Lambda service, and then copy and paste this into this screen:</p>
</p>
You can then hit run and…</p>
</p>
What happened? Well, because it’s a managed</em> instance, the function
doesn’t know what scipy</code> is. It’s not installed on the cloud, it was
installed on your machine…</p>
Layers</h2>
AWS lambda doesn’t pip install ....</code>. Seeing as these run on compute
instances that turn up when needed, and are destroyed when not needed,
with no attached storage, you need to find a way to tell AWS what your
dependencies are, or you’ll just have to write super-pure base Python!
Well, that may not be strictly</em> true. json</code> is built in by default to
every instance</em>, so is boto3</code>, but what about our data science buddies?
numpy</code>, scipy</code> are published by
aws</a> as layers</em>. Layers are bundles of code, that contain the dependencies you need to run the functions you write.
So in this case we can open the ‘layers’ view in AWS and attach these to our function.</p>
</p>
Now that you’ve attached all your dependencies with layers, go ahead and
run your function again.</p>
</p>
Success! So now you know the basics of how to put some Python data
science into practice on AWS Lambda.</p>
This is a companion post to my talk on using data science in AWS lambda.
If you’re keen to know more, and can’t wait for me to write it all up
here. You can get the gist of the whole talks from this repo</a></p>


3 minimal features for my dev.to api wrapper
2020-06-17T00:00:00+00:00
I’ve made 3 very small features for my open source R package wrapping
the dev.to API</a>.</p>
create_new_article</code></h2>
Now that the main requirements for the file to post are stabilising,
I’ve written a quick and dirty function to make a boilerplate article:</p>
create_new_article <-</span>
  function</span></span>(</span>title</span>,</span>
           series</span> =</span> '</span>series'</span></span>,</span>
           tags</span> =</span> '</span>["tag1", "tag2"]'</span></span>,</span>
           file</span> =</span> "</span>"</span></span>)</span></span> {</span>
    boilerplate_frontmatter <-</span>
      glue::</span>glue</span></span>(</span>
        '</span>---\n</span>title: "{title}"\n</span>output: github_document\n</span>series: "{series}"\n</span>tags: {tags}\n</span>---'</span></span>
      )</span></span>

    cat</span></span>(</span>boilerplate_frontmatter,</span> file</span> =</span> file)</span></span>
  }</span>
</span></code></pre>
This will use the glue</code></a> package to put
the strings in the function argument into the right place in the
boilerplate YAML front matter. If then uses cat</code> to either print that
to screen, or to create a new file with it, if a file path is supplied.</p>
main_image</code></h2>
If there is a main_image</code> parameter in the YAML</code> front matter that is
a url of an image, that image will be set as the cover image of the
post. I got this one from a photo by Julian
Dufort</a>
on
Unsplash</a>.
I believe</em> that if you put an unsplash URL into this field that goes
directly to the image</strong> it is within their
license</a>, though if anyone knows
something to the contrary please let me know. It’s the first time I have
used this service, despite hearing about it for years.</p>
Collapse spaces tags</h2>
I recently fooled myself for a good ten minutes into thinking that there
was a problem with my API code yesterday when I kept getting a 422
response to putting a new article up, when in fact it was that I had a
space character in one of my tags. Now the post_new_article</code> function
collapses any spaces it encounters in tags. Achieving this was a breeze
with purrr</code></a> and
stringr</code></a>:</p>
purrr::</span>map</span></span>(</span>file_frontmatter$</span>tags,</span> stringr::</span>str_remove_all,</span> "</span> "</span></span>)</span></span>
</span></code></pre>
This little nugget takes the list of tags, and then maps the function
str_remove_all</code> across all the spaces. This isn’t at all exposed to the
user, as it’s non-negotiable from the API side anyway :)</p>


Webscraping with rvest and themeing ggplot
2020-06-16T00:00:00+00:00
library</span></span>(</span>tidyverse)</span></span>
library</span></span>(</span>pokedex)</span></span>
</span></code></pre>
ggplot</code></a> is the ‘default’ plotting
library in R. It’s a very old package now, but has been kept up-to-date
and is one of the core ‘tidyverse’ packages.
rvest</code></a> is also a tidyverse package that
deals with web scrapping, inspired by equivalents like “beautiful
soup”</a>.</p>
There is a table of hex colour codes used by
Bulbapedia</a>
for each pokemon type. I’d like top be able to use this for plots made
with my pokedex</code> package</a>.</p>
Webscraping with rvest</code></h2>
Get the data</h3>
read_html</span></span>(</span>"</span>https://bulbapedia.bulbagarden.net/wiki/Category:Type_color_templates"</span></span>)</span></span> %>%</span> 
  html_nodes</span></span>(</span>"</span>.wikitable"</span></span>)</span></span> %>%</span>
  .[[</span></span>1</span></span></span>]]</span></span> %>%</span> 
  html_table</span></span>(</span>)</span></span> -></span> pokemon_colour_table
</span></code></pre>
This very simple pipe goes to the url and detects all html nodes with a
class of "wikitable"</code> and puts them in a list. It then takes the first
element (of one in this case), converts it into a table, and assigns it
to a variable pokemon_colour_table</code></p>
Clean the data</h3>
pokemon_colour_table %>%</span>
  janitor::</span>clean_names</span></span>(</span>)</span></span> %>%</span>
  slice</span></span>(</span>1</span>:</span>75</span>)</span></span> %>%</span>
  select</span></span>(</span>-</span>video_game_types_3)</span></span> %>%</span>
  rename</span></span>(</span>type_full</span> =</span> video_game_types,</span> colour</span> =</span> video_game_types_2)</span></span> %>%</span>
  filter</span></span>(</span>type_full !=</span> "</span>"</span></span>)</span></span> %>%</span>
  mutate</span></span>(</span>
    type</span> =</span> tolower</span></span>(</span>str_trim</span></span>(</span>str_remove_all</span></span>(</span>type_full,</span> "</span>color|light|dark|\\</span>:"</span></span>)</span></span>)</span></span>)</span></span>,</span>
    colour_var</span> =</span> case_when</span></span>(</span>
      str_detect</span></span>(</span>type_full,</span> "</span>light"</span></span>)</span></span> ~</span> "</span>light"</span></span>,</span>
      str_detect</span></span>(</span>type_full,</span> "</span>dark"</span></span>)</span></span> ~</span> "</span>dark"</span></span>
    )</span></span>
  )</span></span> %>%</span>
  mutate</span></span>(</span>colour</span> =</span> paste0</span></span>(</span>"</span>#"</span></span>,</span> colour)</span></span>)</span></span> %>%</span>
  select</span></span>(</span>-</span>type_full,</span> type,</span> colour_var,</span> colour)</span></span> -></span> type_colours
</span></code></pre>
Cleaning the data is the more irritating part, as always. First,
janitor::clean_names()</code> does a bunch of sane default things to make
sure our table names are snakecase, with no mad characters and
duplication etc.. Then, as we only want the first part we slice it, and
as we only want the first 2 columns, we drop the third. We then give the
remaining columns sane names, and remove rows that have empty strings.</p>
The meat of the data cleaning comes next, parsing the label column to
get just the type out and convert it to lower case and putting it into a
new column, then conditionally checking if the row is a variant
light/dark hue, or the default, and making a column to represent that.
Finally we convert the colour code to an actual hex string.</p>
Format for ggplot2</code> colour scale</h3>
ggplot2</code> wants the scale as a named list. Making this in a tidy way is
very straightforward.</p>
type_colours %>%</span>
  filter</span></span>(</span>is.na</span></span>(</span>colour_var)</span></span>)</span></span> %>%</span>
  select</span></span>(</span>-</span>colour_var)</span></span> %>%</span>
  mutate</span></span>(</span>colour</span> =</span> set_names</span></span>(</span>colour,</span> type)</span></span>)</span></span> %>%</span>
  pull</span></span>(</span>colour)</span></span> -></span> pokemon_type_scale_colours
</span></code></pre>
In this particular case we select all the values that do not</em> have a
colour_var</code> value, i.e. the defaults, drop the colour_var</code> column, and
set the names of the colour</code> column to the value</em> of the type column.
We have to do this because scale_*_manual()</code> in ggplot will expect a
named list, where the names are the type</code> categorical variable, and the
contents of the list are the hex colour codes for that type. Then when
we pull</code> that column into a list we will have a named list.</p>
Theming</h2>
Add a font with showtext</code></h3>
Keeping the video game flavour, lets also make a quick theme using the a
video game font. We can use
showtext</code></a> to easily add the
“Press Start 2P” font from google fonts.</p>
library</span></span>(</span>"</span>showtext"</span></span>)</span></span>
font_add_google</span></span>(</span>"</span>Press Start 2P"</span></span>)</span></span>
showtext_auto</span></span>(</span>)</span></span>
</span></code></pre>
Then, starting from the theme_minimal</code> we can replace the default font,
and rotate the text labels on the bottom axis.</p>
theme_pokedex</span> </span><-</span> </span>function</span> </span>(</span>)</span></span> {</span>
  theme_minimal</span></span>(</span>)</span></span> %+replace%</span>
    theme</span></span>(</span>
      text</span> =</span> element_text</span></span>(</span>family</span> =</span> "</span>Press Start 2P"</span></span>)</span></span>,</span>
      axis.text.x</span> =</span> element_text</span></span>(</span>angle</span> =</span> -</span>90</span>)</span></span>
    )</span></span>
}</span>

theme_set</span></span>(</span>theme_pokedex</span></span>(</span>)</span></span>)</span></span>
</span></code></pre>
Eeveelutions</h2>
To demonstrate, lets make a simple plot showing the key stats of the
eeveelutions.</p>
pokemon %>%</span> 
  filter</span></span>(</span>evolution_chain_id ==</span> 67</span>)</span></span> %>%</span> 
  select</span></span>(</span>identifier,</span> hp:</span>speed,</span> type_1)</span></span> %>%</span> 
  pivot_longer</span></span>(</span>cols</span> =</span> c</span></span>(</span>hp:</span>speed)</span></span>,</span>
               names_to</span> =</span> "</span>stat"</span></span>)</span></span> %>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> stat,</span> y</span> =</span> value,</span> fill</span> =</span> type_1)</span></span>)</span></span> +</span>
  geom_col</span></span>(</span>)</span></span> +</span>
  facet_wrap</span></span>(</span>. ~</span> identifier)</span></span> +</span>
  scale_fill_manual</span></span>(</span>values</span> =</span> pokemon_type_scale_colours)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>eeveelutions stats"</span></span>)</span></span>
</span></code></pre>
</p>


Writing R packages, fast
2020-06-10T00:00:00+00:00
R packages are great. R users have a rich ecosystem of extensions to
help us doing various things. We have our own integrated</em> package
management system,
CRAN</a>,
and we also have Metacran</a> which gives us an
easy way to work out how popular specific packages are used. R also
makes it ‘easy’ for you to write your own packages! Sort of.</p>
How can I create and R package?</h1>
CRAN suggests making an R package is really
simple</a>.</p>

5.5 How can I create an R package? A package consists of a subdirectory containing a file DESCRIPTION and the subdirectories R, data, demo, exec, inst, man, po, src, and tests (some of which can be missing). The package subdirectory may also contain files INDEX, NAMESPACE, configure, cleanup, LICENSE, LICENCE, COPYING and NEWS.
See section “Creating R packages” in Writing R Extensions, for details. This manual is included in the R distribution, see What documentation exists for R?, and gives information on package structure, the configure and cleanup mechanisms, and on automated package checking and building.
R version 1.3.0 has added the function package.skeleton() which will set up directories, save data and code, and create skeleton help files for a set of R functions and datasets.</p>
</blockquote>
So I just run the function package.skeleton()</code>, and then I just fill in
some R code right? Well, you could. But also, please don’t. Having done
it that way myself the first few times, you can</em> do that, but that
doesn’t mean you should</em>. Luckily R is a programming language, and
people use programming languages to automate things. So obviously people
have built packages the slow way that will help you build your package
easier, better, faster</a>.
We should use those instead!</p>
Starting a project</h1>
You should be using RStudio. It’s an Open
Source</a> IDE for R specifically,
which is free to download</a>. The fastest way to
start your project is to open RStudio, and go to File > New Project > New Directory > R Package</code>. Then give it a name, and make sure to create
a git repository with it.</p>
You now have a Hello World package. It contains a function called
hello</code> in the R</code> directory, with a few comments and notes about short
cuts. You will be able to Build</strong> the package, and you can run a
Check</strong>, which will give you a warning, and run the Test</strong> which
will give you an error.</p>
You’ll also have a few other files and folders. man</code> will hold the
manual, or documentation. The DESCRIPTION</code> gives you a bunch of
metadata, and the NAMESPACE</code> will only have the line
exportPattern("^[[:alpha:]]+")</code>. Don’t worry about that for now, we’ll
fix it in a moment. .Rbuildignore</code> is a bit like the .gitignore</code>. It
will just contain references for R to not</em> use when running Build</strong>.</p>
Setting up a package and managing package structure withusethis</code></h1>
I talked about usethis</code></a> in my last
post</a>,
but didn’t do it nearly the justice it deserves. The best place to start
is Usage</a> and then have a look at the
Reference</a>.</p>
In my projects I tend to use these functions to set up each one:</p>

use_pipe()</code> to allow the package to use pipes from magrittr</code></li>
use_testthat()</code> to set up tests that for the package</li>
use_package()</code> to include a dependency, such as ggplot2</code> or
readr</code></li>
use_vignette()</code> to include long form documentation</li>
use_readme_rmd()</code>/use_readme_md()</code> to create Readmes</li>
use_badge()</code> to fill in those Readmes with badges</li>
</ul>
And then to write my package I tend to use these functions to write it:</p>

use_r()</code> to create a new R file to source</li>
use_test()</code> to create a new test for a specific open R file</li>
use_data()</code> to include data sets in my packages</li>
</ul>
So you can do all these things manually, and maybe the first time you
should. Maybe you should learn that to use a package you need to modify
the Imports</code> field in the DESCRIPTION</code>, and maybe you need to learn
that to make testthat</code> work in your package you need to make a specific
test</code> folder, and then include a testthat.R</code> file to make it work.
However, I also don’t need to implement my own class every time I want
to make an object that holds a data matrix either, and I also don’t need
to know how the interpreter actually</em> works when I run my hello</code>
function. There’s a reason why we don’t write in 1 and 0, and it’s the
same reason I don’t see the value in writing my own NAMESPACE</code> files
any more. I can do it, but it’s a simple boring task that I’ve done a
bunch of times before, and that at this stage I’m actually likely to get
more wrong than the computer. Talking of which…</p>
NAMESPACE</code>, self-documenting code and self-coding documentation</h1>
The NAMESPACE</code> helps your package understand what functions it sources
from what other packages, and what functions it allows to be used. The
R packages book</a> has a chapter
on it that goes into detail, but the key thing to really understand is
how</em> to use it.</p>
As an example, lets run usethis::use_pipe()</code>. This will set-up our
package to allow us to use the %>%</code> operator to chain functions. Our
console tells us:</p>

  ● Run `devtools::document()`
</span></code></pre>

</div>
We also have a new file ./R/utils-pipe.R</code>:</p>
#'</span> Pipe operator
</span>#'</span>
</span>#'</span> See \code{magrittr::\link[magrittr]{\%>\%}} for details.
</span>#'</span>
</span>#'</span> @name</span> %>%
</span>#'</span> @rdname</span> pipe
</span>#'</span> @keywords</span> internal
</span>#'</span> @export</span>
</span>#'</span> @importFrom</span> magrittr %>%
</span>#'</span> @usage</span> lhs \%>\% rhs
</span>NULL</span>
</span></code></pre>
So this is a file that’s all comments? Not quite. This is a Roxygen
comment block with #'</code>, and it has special powers. It won’t actually be
executed if this file is sourced, but if we document</em> our package, the
process will read these comment blocks, and the @</code> tags, and populate
out the documentation in ./man</code>. As we got prompted though, we need to
run the documentation process manually, so lets do that with
devtools::document()</code>.</p>

  Updating</span></span> making.packages documentation</span>
First</span></span> time using roxygen2. Upgrading automatically...</span>
Writing</span></span> NAMESPACE</span>
Loading</span></span> making.packages</span>
Writing</span></span> NAMESPACE</span>
Writing</span></span> pipe.Rd</span>
</span></code></pre>

</div>
Now we have some changes. Our NAMESPACE</code> LOOKS LIKE THIS:</p>
#</span> Generated by roxygen2: do not edit by hand
</span>
export</span></span>(</span>"</span>%>%"</span></span>)</span></span>
importFrom</span></span>(</span>magrittr,</span>"</span>%>%"</span></span>)</span></span>
</span></code></pre>
and we have a new ./man/pipe.Rd</code> file:</p>
% Generated by roxygen2:</span> do not edit by hand
% Please edit documentation in</span> R/</span>utils-</span>pipe.R
\name{</span>\%></span>\%}</span>
\alias{</span>\%></span>\%}</span>
\title{</span>Pipe operator}</span>
\usage{</span>
lhs \%></span>\% rhs
}</span>
\description{</span>
See \code{</span>magrittr::</span>\link[</span></span>magrittr</span></span>]</span></span>{</span>\%></span>\%}</span>}</span> for details.
}</span>
\keyword{</span>internal}</span>
</span></code></pre>
What’s happened? A few things:</p>

Roxygen was associated with our package build to parse and process
the comments</li>
Roxygen re-wrote the NAMESPACE</code></li>
Roxygen read the Roxygen comments in the ./R/utils-pipe.R</code></li>
Roxygen used those special comments to put an import and an export
line into the NAMESPACE</code>

The export</code> will allow users to use the %>%</code> when they load
our package</li>
The importFrom</code> will allow us to use the %>%</code> in our own code
in the package itself</li>
</ul>
</li>
Roxygen populated the manual page for the function so we can see it
in the RStudio help.</li>
</ol>
Compare the two comment blocks now. What are the differences?</p>
Roxygen comments</h2>
#'</span> Pipe operator
</span>#'</span>
</span>#'</span> See \code{magrittr::\link[magrittr]{\%>\%}} for details.
</span>#'</span>
</span>#'</span> @name</span> %>%
</span>#'</span> @rdname</span> pipe
</span>#'</span> @keywords</span> internal
</span>#'</span> @export</span>
</span>#'</span> @importFrom</span> magrittr %>%
</span>#'</span> @usage</span> lhs \%>\% rhs
</span>NULL</span>
</span></code></pre>
Roxygenised .Rd</code></h2>
% Generated by roxygen2:</span> do not edit by hand
% Please edit documentation in</span> R/</span>utils-</span>pipe.R
\name{</span>\%></span>\%}</span>
\alias{</span>\%></span>\%}</span>
\title{</span>Pipe operator}</span>
\usage{</span>
lhs \%></span>\% rhs
}</span>
\description{</span>
See \code{</span>magrittr::</span>\link[</span></span>magrittr</span></span>]</span></span>{</span>\%></span>\%}</span>}</span> for details.
}</span>
\keyword{</span>internal}</span>
</span></code></pre>
So the first big one is formatting. The first one looks (mostly) human
readable, while the second one has a lot more cruft. This cruft is
basically LaTeX. The order has been changed around and few of the labels
are a little different, but there is another big change. @export</code> and
@importFrom</code> aren’t in the documentation! Those were only</em> included
to help Roxygen populate the NAMESPACE</code> file. So you can see that those
two tags are actually amongst the most important things to include in
documenting other functions. If you check in the DESCRIPTION</code> file we
now also</em> have a new Imports</code> field in the YAML with magrittr</code> being
the only entry. Did you notice that there was no actual</em> R code in this
function? It actually explicitly contained NULL</code>. All of this was
purely generated from the documentation, but now we actually have a new
piece of fundamental functionality in our package. Weird huh?</p>
The package skeleton’s connected to the comment bone. The comment bone’s connected to the sinew imports.</h1>
So now you’re all prepared to write your own package. You know how to
start a new package project, how to set it up with usethis</code> and how to
document functions with Roxygen. Hold up though. That thing about not
having to do the basically the same fiddly boring job multiple times in
each package? I really meant it. The cool R kids don’t even write their
own Roxygen any more.</p>
Go back to your ./R/hello.R</code> file, and ditch those boring vanilla #</code>
comments. Now source the function into your global environment so you
can run it interactively. Now install.packages("sinew")</code>, and run
sinew::makeOxygen(hello)</code>.</p>
#'</span> @description</span> FUNCTION_DESCRIPTION
</span>
#'</span> @return</span> OUTPUT_DESCRIPTION
</span>#'</span> @details</span> DETAILS
</span>#'</span> @examples</span> 
</span>#'</span> \dontrun{
</span>#'</span> if(interactive()){
</span>#'</span>  #EXAMPLE1
</span>#'</span>  }
</span>#'</span> }
</span>#'</span> @rdname</span> hello
</span>#'</span> @export</span> 
</span></span></code></pre>
So sinew</a> has built us a template for
Roxygen, and put the function name in? Cute, but I don’t need that to
write one word for me. OK, how about something a bit more real world.
I’ve been making a wrapper for the dev.to
API</a>
(using all these methods). If I go get my post_new_article</code> function
and run sinew::makeOxygen(post_new_article)</code>:</p>

  #'</span> @title</span> FUNCTION_TITLE
</span>#'</span> @description</span> FUNCTION_DESCRIPTION
</span>#'</span> @param</span> file</span> PARAM_DESCRIPTION
</span>#'</span> @param</span> key</span> PARAM_DESCRIPTION, Default: NA
</span>#'</span> @return</span> OUTPUT_DESCRIPTION
</span>#'</span> @details</span> DETAILS
</span>#'</span> @examples</span> 
</span>#'</span> \dontrun{
</span>#'</span> if(interactive()){
</span>#'</span>  #EXAMPLE1
</span>#'</span>  }
</span>#'</span> }
</span>#'</span> @seealso</span> 
</span>#'</span>  \code{\link[rmarkdown]{yaml_front_matter}},\code{\link[rmarkdown]{render}}
</span>#'</span>  \code{\link[readr]{read_file}}
</span>#'</span>  \code{\link[stringr]{str_remove}}
</span>#'</span>  \code{\link[glue]{glue}}
</span>#'</span>  \code{\link[httr]{POST}},\code{\link[httr]{add_headers}}
</span>#'</span> @rdname</span> post_new_article
</span>#'</span> @export</span> 
</span>#'</span> @importFrom</span> rmarkdown yaml_front_matter render
</span>#'</span> @importFrom</span> readr read_file
</span>#'</span> @importFrom</span> stringr str_remove
</span>#'</span> @importFrom</span> glue glue
</span>#'</span> @importFrom</span> httr POST add_headers
</span></span></code></pre>

</div>
This is my output. Sinew inspects my function, and then derives</em> the
functions I need to import and adds them to the Roxygen comment block.
It also links the functions in the @seealso</code> which will show up in the
docs. Remember that these now get automatically</em> added to the
NAMESPACE</code> and DESCRIPTIONS</code> where they need to be, each time I
document my package. It’s also understood what arguments my function
takes, and populated them with the defaults, if present. Now, why would
you bother writing this by hand any more?</p>
TL;DR</h1>

Start your package in the RStudio IDE as a new project</li>
Set up your package with usethis</code></li>
Populate your Roxygen comments with sinew</code></li>
Profit!</li>
</ol>


Building my first Django project with a Database
2020-06-05T00:00:00+00:00
I’m working through Django for Beginners by William S. Vincent</a>. Chapter 4 starts using a database to act as a data store for a Message Board app.</p>
This seemed very related to what my buddies do with Laravel, where migrations are run when the data Model in the backend needs to be changed. I was actually a little surprised that at no point did I need to write any SQL though. As someone who has spent their professional life asking databases for info, I was actually a tiny bit disappointed 😜. </p>
I’m starting to become curious about this manage.py</code> file though. All I have in my workspace to represent it is:</p>
#</span>!/usr/bin/env python
</span>"""</span>Django's command-line utility for administrative tasks."""</span></span>
import</span> os</span></span></span>
import</span> sys</span></span></span>


def</span></span> main</span></span></span>(</span></span>)</span></span>:</span></span>
    os</span>.</span>environ</span></span>.</span></span>setdefault</span></span></span></span>(</span>'</span></span></span>DJANGO_SETTINGS_MODULE'</span></span></span>,</span> '</span></span></span>blog_project.settings'</span></span></span>)</span></span>
    try</span>:</span></span>
        from</span></span> django</span>.</span>core</span>.</span>management</span></span> import</span></span></span></span></span> execute_from_command_line</span></span>
    except</span> ImportError</span> as</span></span> exc</span></span>:</span></span>
        raise</span> ImportError</span></span></span></span>(</span>
            "</span></span></span>Couldn't import Django. Are you sure it's installed and "</span></span></span>
            "</span></span></span>available on your PYTHONPATH environment variable? Did you "</span></span></span>
            "</span></span></span>forget to activate a virtual environment?"</span></span></span>
        )</span></span> from</span></span></span> exc</span></span></span>
    execute_from_command_line</span></span></span></span>(</span>sys</span>.</span>argv</span></span>)</span></span>


if</span> __name__</span></span> ==</span> '</span></span></span>__main__'</span></span></span>:</span></span>
    main</span></span></span></span>(</span>)</span></span>
</span></code></pre>
So it’s obviously getting some more code under the hood, but I’m starting to think it has super powers. It starts projects and apps, it makes users, it runs migrations and tests as well as the local server. For something that hasn’t even really been mentioned other than a line of text to type into the CLI my curiosity is definitely rising. Unfortunately python manage.py makecoffee' is the only thing that hasn't worked so far. What are your favourite powers of </code>manage.py`?</p>


Building my first Django project with pages
2020-06-03T00:00:00+00:00
I’m working through Django for Beginners by William S. Vincent</a>. Chapter 3 starts using templates and introduces Class-Based Views.</p>
There is a section introducing Class-Based views, and then mentioning older Function-Based views. It was interesting to see this evidence of the framework progressing, though it’s not so surprising for something that’s a decade or so old!</p>
The templating syntax was really neat. I’d seen very similar in Hugo and Jekyll, and so the idea of having a kind of ‘placeholder’ with “{% block content %}{% endblock content %}</code> was nice to use. I’ve admittedly done very little webdev though. I was curious what other approaches might be common alternatives? If you have an example please let me know in the comments!</p>
Something that was curious to me was the inversion of structure in django vs hugo.</p>
In hugo, I’ve very regularly used partials such as in the SatRdays theme here</a>. Here, I have a single home page with my headers and footers, and ‘inject’ a partial into it with {{ partial "nav.html" . }}</code>. In django I feel like it kind of works the other way. We create a project, and then the equivalent to a hugo partial would actually be a django app within the project? Then, each app shares top level content through lines like {% extends 'base.html' %}</code> within each apps template?</p>
It seems like it might be more brittle to me, but maybe also more powerful? There’s more duplicate lines to get wrong, but also more opportunities to vary what it does?</p>
What do veteran djangians(is that right?) think? Does anyone use both Hugo and Django, but for different projects and for specific reasons?</p>


Testing my dev.to API package with testthat, webmockr and vcr
2020-06-03T00:00:00+00:00
I’ve been working on an open source R package wrapping the dev.to
API</a>. After a bit of prototyping
the core feature and then some iterative development to make it a bit
more useful to a user, it’s starting to settle into something that is
worth testing\*.</p>
Testing in R</h2>
Sometimes people seem a little surprised that this is something R users/
Data Scientists even think about, but R is a programming language, just
like all the others, except different, just like all the others. Testing
in modern tidyverse R is usually accomplished by the
testthat</code></a> package. If you are more
familiar with testing in other languages, this quote from the Overview
might help:</p>

testthat draws inspiration from the xUnit family of testing packages, as well as from many of the innovative ruby testing libraries, like rspec, testy, bacon and cucumber.</p>
</blockquote>
A note on usethis</code></h3>
In R we have a rich ecosystem of developer tools to help us make
packages. testthat</code> is one. Another one is usethis</code>. See what they did
there? Anyway, for the purpose of this article, you just need to know
that a package called usethis</code> helps us set up things for us to make
development easier. In the future I might write something more about it
¯_(ツ)_/¯.</p>
testthat</code></h2>
Setting up testthat</code></h3>
Assuming that you have a well formatted R Package structure, you can
easily enable a testing framework and all the boiler plate you might
need with one line:</p>
usethis::</span>use_testthat</span></span>(</span>)</span></span>
</span></code></pre>
Which will do a few things for you and tell you what it’s doing.

  ✔</span></span> Setting active project to '</span>/Users/davidparr/Documents/example.testthat'</span></span></span>
✔</span></span> Adding '</span>testthat'</span></span> to Suggests field in DESCRIPTION</span>
✔</span></span> Creating '</span>tests/testthat/'</span></span></span>
✔</span></span> Writing '</span>tests/testthat.R'</span></span></span>
●</span></span> Call `</span>use_test</span>(</span>)</span>` to initialize a basic test file and open it for editing.
</span></span></span></span></code></pre>

</div>
</p>
From there, we can use usethis::use_test()</code>. If you have a file open in
RStudio which is an *.R</code> file containing function definitions it will
then make a new test file for you:</p>

  ✔</span></span> Increasing '</span>testthat'</span></span> version to '</span>>= 2.1.0'</span></span> in DESCRIPTION</span>
✔</span></span> Writing '</span>tests/testthat/test-hello.R'</span></span></span>
●</span></span> Modify '</span>tests/testthat/test-hello.R'</span></span></span>
</span></code></pre>

</div>
The contents of that test file will be silly boilerplate, so now it’s
time to do some real work.</p>
Writing a testthat</code> test</h3>
test_that</span></span>(</span>"</span>multiplication works"</span></span>,</span> {</span>
  expect_equal</span></span>(</span>2</span> *</span> 2</span>,</span> 4</span>)</span></span>
}</span>)</span></span>
</span></code></pre>
This is the boilerplate. If you run it, you might be surprised that
nothing happens! Well, actually a lot of stuff happens, but it doesn’t
really tell you about it by design. If you make a test that isn’t going
to pass however…</p>
test_that</span></span>(</span>"</span>maths works"</span></span>,</span> {</span>
  expect_equal</span></span>(</span>2</span> *</span> 2</span>,</span> 5</span>)</span></span>
}</span>)</span></span>
</span></code></pre>

  ## Error: Test failed: 'maths works'
## * <text>:2: 2 * 2 not equal to 5.
## 1/1 mismatches
## [1] 4 - 5 == -1
</span></code></pre>

</div>

An error! Just what we wanted! so this is the basis of testing with
testthat</code>. Write a test_that()</code> function, which has a name, and then a
block of expectations to check against.</p>
webmockr</code></h2>
So this works fine for traditional unit tests, where we can give
discrete calculations, or check that a given function gives a specific
output end to end, where we control both the test, but also the function
as a whole. But what if we’re reliant on some ‘external’ process that we
might not control. The dev.to API for instance? I’m not employed by
dev.to (through I am looking for a new
opportunity</a>),
so I don’t get to play around inside the API system, but I do need to
prove that any code I write will behave the way I want it to based on
their API requests and responses. A simple way to prove this is to
mock</em> their service (i.e. impersonate it, not tell it it’s silly). This
is where webmockr</code></a> comes in.
webmockr</code> is an:</p>

“R library for stubbing and setting expectations on HTTP requests”</p>
</blockquote>
Perfect. Let’s write something using both testthat</code> and webmockr</code>.</p>
Writing a webmockr</code> test</h3>
In webmockr</code> we make stubs</em>. These are fake, minimal objects that are
similar to test fixtures. We know their properties, because we made
them, and we want to make sure that any functions we write interact with
these objects in a predictable way. Another way of looking at them is as
a fake API. They look like an API, with responses and status codes, but
they only exist in our test suite. This means I don’t need to bombard
dev.to with requests any more to make sure I haven’t broken anything.</p>
webmockr::</span>enable</span></span>(</span>adapter</span> =</span> "</span>httr"</span></span>)</span></span>

webmockr::</span>stub_registry_clear</span></span>(</span>)</span></span>

webmockr::</span>stub_request</span></span>(</span>"</span>get"</span></span>,</span> "</span>https://dev.to/api/users/me"</span></span>)</span></span> %>%</span>
  webmockr::</span>to_return</span></span>(</span>body</span> =</span> "</span>success!"</span></span>,</span> status</span> =</span> 200</span>)</span></span>

my_user <-</span> dev.to.ol::</span>get_my_user</span></span>(</span>)</span></span>

test_that</span></span>(</span>"</span>stubbed response is 200"</span></span>,</span> {</span>
  expect_is</span></span>(</span>my_user,</span> "</span>response"</span></span>)</span></span>
  expect_equal</span></span>(</span>my_user$</span>status_code,</span> 200</span>)</span></span>
}</span>)</span></span>
</span></code></pre>
This code first of all sets up our test file to understand that requests
will be sent as if from the httr</code> package. It then clears the registry,
just to make sure nothing is left in a cache. It then populates the now
empty cache with a new stub. This stub will respond to a GET</code> request
to the URL, and will return a simple text body, and a 200 status code.
The function I want to test is then run, which in this environment hits
the stub</em>, not the real api. The object that is returned is then
checked by test_that</code> to make sure it is a response</code> type object, and
that is has a status code that has the value 200.</p>
Good enough, but is it actually enough?</h3>
This proves a few, specific things. That the function returns a
response</code> that has a 200 status code if it trys to GET</code> from that
specific URL. However, APIs actually return quite a lot of information
by default and maybe I care about more things than a 200 status code.
They can also have quite complex structures, so using this method to
make a fake response could get very</em> awkward if I am trying to make a
realistic response. Also, what if the structure of the api changes
underneath us? It is in beta after all. A big change would mean all
those carefully written pipes would have to be rewritten by me every
time. Blergh. Luckily we have a solution for that too!</p>
vcr</code></h2>
vcr</code></a> does not stand for Very Cool Responses</code>, but I think it should. From it’s own description:</p>

Record HTTP calls and replay them</p>
</blockquote>
It’s an R port of a ruby gem of the same name, this package allows you
to ‘record’ the response of an API to a YAML file ‘cassette’. You can
then ‘play’ the ‘cassette’ back during the test as if the API was being
actually called. If you’re still not sure where the name comes from,
then you might be a little to young to get the
reference</a>.</p>
Setting up vcr</code></h3>
vcr</code> has a few things it needs in a project to run, and though it
doesn’t have its own</em> entry in usethis</code>, it does have it’s own
set-up function in a similar style:</p>
vcr::</span>use_vcr</span></span>(</span>)</span></span>
</span></code></pre>

  ◉</span></span> Using package: vcr.example  </span>
◉</span></span> assuming fixtures at: tests/fixtures  </span>
✓</span></span> Adding vcr to Suggests field in DESCRIPTION  </span>
✓</span></span> Creating directory: ./tests/testthat  </span>
◉</span></span> Looking for testthat.R file or similar  </span>
✓</span></span> tests/testthat.R: added  </span>
✓</span></span> Adding vcr config to tests/testthat/helper-vcr.example.R  </span>
✓</span></span> Adding example test file tests/testthat/test-vcr_example.R  </span>
✓</span></span> .gitattributes: added  </span>
◉</span></span> Learn more about `</span>vcr</span></span>`</span></span>: https://books.ropensci.org/http-testing</span>
</span></code></pre>

</div>
From there, we can use the normal testthat</code> flow. Here’s an example
using the POST</code> to write a new article.</p>
test_that</span></span>(</span>"</span>post new article"</span></span>,</span> {</span>
  vcr::</span>use_cassette</span></span>(</span>"</span>post_new_article"</span></span>,</span> {</span>
    new_article <-</span> dev.to.ol::</span>post_new_article</span></span>(</span>file</span> =</span> "</span>./test.Rmd"</span></span>)</span></span>
  }</span>)</span></span>

  expect_is</span></span>(</span>new_article,</span> "</span>response"</span></span>)</span></span>
  expect_equal</span></span>(</span>new_article$</span>status_code,</span> 201</span>)</span></span>
}</span>)</span></span>
</span></code></pre>
Well, that’s an easy change! The only difference from the first test is
that we have wrapped the function we are testing in a use_cassette</code>
block, inside the test_that</code> block. Now, the first time this function
is run, you get
this</a>.
A huge YAML file that describes the response of the actual API. Now,
every time the test is run, that cassette will get loaded as the ‘mock’,
and it’s so much more developed than our stub! We can test against
anything we want in the response, and even better, the response is
totally human readable.</p>
What about changes? What happens if you make a change to the data you
use to test the function that invalidates the cassette</code>? What if the
dev.to spec changes? Easy, all you do is delete the test and re-run. The
function will then go and get a new response, and populate the file
again. Your tests then run against the new file. You can even commit
these to version control. Then you can tell exactly when an API change
occurred, and what was different afterwards.</p>
Thanks Scott</h2>
Both the webmockr</code> and vcr</code> packages are being maintained by @sckott,
who is an active writer here on dev.to. and I think he’s really worth a
follow. He also works on ROpenSci</a>, which I
think is also a really cool project. If you are working with R on a
scientific/research project you should be extra interested.</p>

Yes, I know that TDD exists. IMHO: No, I don’t think it’s a bad thing, but also no, I don’t think it’s always something to use everywhere all the time.</li>
</ul>


Building my first Django project
2020-06-01T00:00:00+00:00
I’m working through Django for Beginners by William S. Vincent</a>. The second chapter gets us into developing our first app. It simply displays a single page with an unformatted Hello World on it, but I really appreciated how the project is so fully fleshed out end to end. You start a project with a clean pipenv</code>, and make the files and folders through the CLI. The django framework provides some really nice methods to ‘boilerplate’ the multiple files needed in each project, and having this introduced at the start and then fill in the details later when you are hands on is definitely the best way to go.</p>
Introducing migrations and the url-view-model[-template] core concepts was pretty simple in practice, though a little confusing initially. “I’ve just made a bunch of files with this single startapp</code> command, what do they all do?” was admittedly my first reaction. </p>
The best part of the way this book is set up is the definition of done used in the core workflow. A project isn’t done unless it has a commit history, is hosted on GitHub and deployed on Heroku. I’m really a fan of approaching deployment in Chapter 2, just like version control and dev environments in Chapter 1. These concepts are your friends, not your monsters :)</p>
Incidentally, having only been aware of Heroku in passing, I really like the PAAS set up. Very usable, the cli tools make the whole thing a doddle and the end satisfaction of being able to point at something that’s actually on the web and say ‘I did that!’ is always a little more satisfying than ‘It works on my machine?’.</p>


Setting up a Django dev environment
2020-05-27T00:00:00+00:00
I’m working through Django for Beginners by William S. Vincent</a>. The first part is about setting up your dev environment. The book is setup to start for a complete</em> beginner level, which is really nice. It starts from introducing the basics of the CLI for both Mac and Windows systems, and then some simple terminal commands with cd</code>, ls</code> etc.</p>
It then goes on to help with installation of Python3, git</code>, pipenv</code> and django</code>, all great stuff for getting started, plus it set’s a great baseline for the rest of the book.</p>
For anyone who has worked at all with terminals, version control, isolating environments and code editors, 90% of this will be already done on the machine, but I still think it’s worth putting in. Having it laid out for you at the start of your programming experience that version control and the terminal are part of the process I think helps with the hump I’ve seen in more experienced beginners where git</code> and pipenv</code> are Yet Another Thing To Learn and so get pushed back in the learning pathway more out of fear than anything else.</p>
Personally, I zipped through the setup in a few minutes. Hello World django app next!</p>
Has anyone else seen the ‘git is yet another thing to learn’ hump? Maybe even had it? How did you overcome it?</p>


I made my dev.to content into a website to find a new job
2020-05-25T00:00:00+00:00
After discovering this post</a> and this post</a>, I decided to use this tooling to get a very simple job done to help me look for work.</p>
Context</h2>
I’ve been using dev.to a little over the last year, and a lot more over the last month. It makes sense for me to use this content in my job hunt. I also had a languishing personal page. Originally made as a little experiment with hugo, it hadn’t seen love in a while, and due to some jankey usage of high quality photos performance wise it was pretty bad. I’d been meaning to overhaul it for a while, and when the dev.to x stackbit collab came to my attention, it was a perfect fit.</p>
Goals</h2>

Use the content I’ve written, and will be continuing to write, to prove I know at least a few things about programming.</li>
Get that content wrapped up neatly with a contact page, a statement that I’m looking for work, and a brief about me</li>
Use my daveparr.info domain</li>
Make the website more performant than it’s current iteration</li>
Use a clean, simple, slightly professional theme</li>
</ol>
Method</h2>

Read the instructions in @ben ’s post</a></li>
Go through the creation flow</li>
Assign the domain in netlify (I had a netlify hosted JAMstack previously, and my domain was already loaded in there. YMMV)</li>
Clone the project @stackbit created from GitHub, and follow the readme</a> they left there to help you authenticate correctly</li>
Update the boilerplate stuff</a></li>
Push the changes back to GitHub</li>
Profit</li>
</ol>
Hiccup</h2>
I was a little surprised when my local copy suddenly got all my blog files that I then just commited</a>, but that seemed to be ok. Pretty sure I’ve not broken anything :)</p>
The End</h2>
Goals achieved, faster website, with better styling, that will be kept more up to date.</p>
P.S.</h2>
If you are reading this on</em> my website</a>, this is all actually managed through a headless CMS called dev.to</a>. You can use the link to my profile on the left of this article. If you are reading this on dev.to you can see my running website at daveparr.info</a>.</p>


Posting straight from Rmd to DEV.TO (for real this time)
2020-05-24T00:00:00+00:00
I’ve spent a little time fleshing out my open source R package</a> to post from .Rmd</code>
straight to dev.to.</p>
Update from V1</h2>
The biggest difference from the first version</a> is
that there is now a single function to move straight from an .Rmd</code> on
disk to a post on dev.to. In V1 the user still had to:</p>

Write an .Rmd</code></li>
knit</code> to github_document</code></li>
post_new_article</code> using my dev.to.ol</code> package</li>
De-duplicate the title</li>
(Optionally) Add meta-data</li>
</ol>
With this new function the workflow is:</p>

Write an .Rmd</code></li>
post_new_article</code> using my dev.to.ol</code> package</li>
</ol>
🎉</p>
#'</span> @title</span> Post a markdown file to dev.to
</span>#'</span> @description</span> Create a new post from an .Rmd.
</span>#'</span> @param</span> file</span> The path to the file
</span>#'</span> @param</span> key</span> Your API key, Default: NA
</span>#'</span> @return</span> The response
</span>#'</span> @details</span> Will look for an api key in the `.REnviron` file. Seems to check if the body is identical to a previous article and error if so with `"Body markdown has already been taken"`.
</span>#'</span> The following YAML arguments are read from the file YAML frontmatter if present:
</span>#'</span> \describe{
</span>#'</span>   \item{title}{A character string}
</span>#'</span>   \item{series}{A character string}
</span>#'</span>   \item{published}{A boolean}
</span>#'</span>   \item{tags}{list of character strings: \code{["tag1", "tag2"]}}
</span>#'</span> }
</span>#'</span>
</span>#'</span> The default table output method renders a very large print code block.
</span>#'</span> The workaround is to use  \code{\link[knitr]{kable}}.
</span>#'</span>
</span>#'</span> @examples</span>
</span>#'</span> \dontrun{
</span>#'</span> if(interactive()){
</span>#'</span>  post_new_article("./articles/my_article.Rmd")
</span>#'</span>  }
</span>#'</span> }
</span>#'</span> @seealso</span>
</span>#'</span>  \code{\link[rmarkdown]{yaml_front_matter}},\code{\link[rmarkdown]{render}}
</span>#'</span>  \code{\link[readr]{read_file}}
</span>#'</span>  \code{\link[stringr]{str_remove}}
</span>#'</span>  \code{\link[glue]{glue}}
</span>#'</span>  \code{\link[httr]{POST}},\code{\link[httr]{add_headers}},\code{\link[httr]{content}}
</span>#'</span> @rdname</span> post_new_article
</span>#'</span> @export</span>
</span>#'</span> @importFrom</span> rmarkdown yaml_front_matter render
</span>#'</span> @importFrom</span> readr read_file
</span>#'</span> @importFrom</span> stringr str_remove
</span>#'</span> @importFrom</span> glue glue
</span>#'</span> @importFrom</span> httr POST add_headers content
</span>
post_new_article <-</span>
  function</span></span>(</span>file</span>,</span>
           key</span> =</span> NA</span>)</span></span> {</span>
    check_file <-</span> is_postable_Rmd</span></span>(</span>file)</span></span>

    if</span> (</span>check_file)</span> {</span>
      file_frontmatter <-</span> rmarkdown::</span>yaml_front_matter</span></span>(</span>file)</span></span>

      output_path <-</span> rmarkdown::</span>render</span></span>(</span>'</span>./data/test.Rmd'</span></span>,</span>
                                       output_format</span> =</span> '</span>github_document'</span></span>,</span>
                                       output_dir</span> =</span> getwd</span></span>(</span>)</span></span>)</span></span>

      file_string <-</span> readr::</span>read_file</span></span>(</span>output_path)</span></span> %\u003e%
        stringr::</span>str_remove</span></span>(</span>glue::</span>glue</span></span>(</span>"</span>{title}\n</span>================\n</span>\n</span>\n</span>"</span></span>,</span>
                                       title</span> =</span> file_frontmatter$</span>title)</span></span>)</span></span>

      response <-</span> httr::</span>POST</span></span>(</span>
        url</span> =</span> "</span>https://dev.to/api/articles"</span></span>,</span>
        httr::</span>add_headers</span></span>(</span>"</span>api-key"</span></span> =</span> api_key</span></span>(</span>key</span> =</span> key)</span></span>)</span></span>,</span>
        body</span> =</span> list</span></span>(</span>
          article</span> =</span> list</span></span>(</span>
            title</span> =</span> file_frontmatter$</span>title,</span>
            series</span> =</span> file_frontmatter$</span>series,</span>
            published</span> =</span> file_frontmatter$</span>published,</span>
            tags</span> =</span> file_frontmatter$</span>tags,</span>
            body_markdown</span> =</span> file_string
          )</span></span>
        )</span></span>,</span>
        encode</span> =</span> '</span>json'</span></span>
      )</span></span>
      httr::</span>content</span></span>(</span>response)</span></span>
    }</span> else</span> {</span>
      message</span></span>(</span>attr</span></span>(</span>check_file,</span> "</span>msg"</span></span>)</span></span>)</span></span>
    }</span>
  }</span>
</span></code></pre>
Notable changes</h2>
Extract YAML</code> frontmatter and render from .Rmd</code> with rmarkdown</code></a></h3>
The function will now accept an .Rmd</code> file directly which it renders to
the correct markdown output internally. This was the biggest part
missing from V1, and was easy to do with rmarkdown::render</code>. A great
side-effect is that I can also access the YAML</code> frontmatter of the
.Rmd</code>, which I can use to populate the meta-data such as the series the
post is in, the tags the post is relevant to, and the published status.</p>
### Check file is suitable with
assertthat</code></a></p>
Another benefit is that there is now a concrete object I can check for
correctness. I’ve adopted the assertthat</code> package to help me, and most
of the work was done in this
commit</a>.
The workhorse function is
is_postable_Rmd</code></a>:</p>
is_postable_Rmd</span> </span><-</span> </span>function</span></span>(</span>file</span>)</span></span> {</span>
    assertthat::</span>see_if</span></span>(</span>
    assertthat::</span>is.readable</span></span>(</span>file)</span></span>,</span>
    assertthat::</span>has_extension</span></span>(</span>file,</span> "</span>Rmd"</span></span>)</span></span>
    )</span></span>
}</span>
</span></code></pre>
Using assertthat::see_if</code> was really useful. It allowed me to check for
multiple conditions, and if they were both passed, it returns TRUE</code>.
However, if they did not pass, it returns FALSE</code> as well as a message
attribute</em>. This meant that I could very trivially plumb that back into
the main function for user feedback as to what is the problem with the
line message(attr(check_file, "msg"))</code>. Additionally, as I add criteria
to the function I know they will all return human, well formatted error
messages. Great work assertthat</code>!</p>
Deduplicate the title with glue</code></a> and stringr</code></a></h3>
One of the issues with the original approach was the call to render will
generate an .md</code> with a title in the file</em>, but the dev.to api wants
the title as a separate part of the call</em>. stringr</code> and glue</code> to the
rescue! Using glue</code> is basically a super powered paste. It does nice
things to insert named string variables into a templated string. I use
this to make the string that is what I want to remove</em> from the
body_markdown</code> object, and then do the actual removal with
str_remove</code>.</p>
Next steps</h2>
Images. The real power of Rmarkdown comes from turning code into
notebooks populated with graphics representing your data science work. I
have not yet found an api endpoint for
images</a>.
The POST looks like it will allow a cover image to be included, but on
the face of it I don’t think it will allow images to be inserted at
arbitrary points in the article text. Still, I’ll start there and see
where it goes!</p>


Posting from .Rmd to dev.to
2020-05-20T00:00:00+00:00
I’ve started making an R package called
dev.to.ol</code></a>. Dev.to has an api
which is in beta</a>, which I’m using to power
the package.</p>
Prototype</h2>
The first function was really just to make sure I could use the api at
all. It gets all the articles for the authenticated user. What’s my most
recent articles title?</p>
library</span></span>(</span>dev.to.ol)</span></span>
dev.to.ol::</span>get_users_articles</span></span>(</span>)</span></span>[[</span></span>1</span></span></span>]]</span></span>$</span>title
</span></code></pre>

  ## Using DEVTO in .Reinviron

## [1] "Tidy Tuesday and space to learn"
</span></code></pre>

</div>
So how does this function work?</p>
#'</span> @title</span> get the authenticated users articles
</span>#'</span> @description</span> Provides lots of info on your users articles
</span>#'</span> @param</span> key</span> the api you have set up on DEV.TO
</span>#'</span> @return</span> article stuff
</span>#'</span> @details</span> if no key is supplied, will check for key named DEVTO in `.Renviron`
</span>#'</span> @examples</span>
</span>#'</span> \dontrun{
</span>#'</span> if(interactive()){
</span>#'</span>  get_users_articles("my_api_key")
</span>#'</span>  }
</span>#'</span> }
</span>#'</span> @seealso</span>
</span>#'</span>  \code{\link[httr]{content}},\code{\link[httr]{GET}},\code{\link[httr]{add_headers}}
</span>#'</span> @rdname</span> get_users_articles
</span>#'</span> @export</span>
</span>#'</span> @importFrom</span> httr content GET add_headers
</span>
get_users_articles</span> </span><-</span> </span>function</span></span>(</span>key</span> =</span> NA</span>)</span></span> {</span>
  httr::</span>content</span></span>(</span>httr::</span>GET</span></span>(</span>url</span> =</span> "</span>https://dev.to/api/articles/me"</span></span>,</span>
                          httr::</span>add_headers</span></span>(</span>"</span>api-key"</span></span> =</span>
                                              if</span> (</span>!</span>is.na</span></span>(</span>key)</span></span>)</span> {</span>
                                                key
                                              }</span> else</span> {</span>
                                                message</span></span>(</span>"</span>Using DEVTO in .Renviron"</span></span>)</span></span>
                                                Sys.getenv</span></span>(</span>x</span> =</span> "</span>DEVTO"</span></span>)</span></span>
                                              }</span>)</span></span>)</span></span>)</span></span>
}</span>
</span></code></pre>
Very simply</p>

It uses httr</code> to do most of the work through a GET</code> request.</li>
It allows a user to supply their own api key as an argument.</li>
If the user has left that argument as the default NA</code>, it will use
the environmental variable named DEVTO</code>. This can be set with an
.Renviron</code> file at the project or user level that looks like this:</li>
</ul>

DEVTO="obviouslynotmyrealkey"
</span></code></pre>
Motivation</h2>
So as R users, we have a great tool baked into our language: rmarkdown.
I use it for nearly everything. We also have some great tools to magnify
it’s power, such as R Notebooks, blogdown, package down and distill.
Some great R people are making the effort to put content here such as
Julia Silge (@juliasilge) and Colin Fay (@colinfay), though they are
already well established bloggers, and are publishing from their main
website through RSS. It’s a great solution for them but personally I was
looking at dev.to as a great way to not</em> have to have a whole website.
So what do you do if you want to publish a blog, but don’t want to
actually maintain a full website? What if you also want to be able to be
part of the great dev.to community? What if you’re both of those things
and also find you have a large volume of time on your hands? You write a
package to put .Rmd</code>s onto dev.to as simply as possible.</p>
Workflow</h2>
My current process is this:</p>

Write an Rmarkdown</li>
Render to github_document</code> style markdown</li>
Open the markdown file I just made</li>
Copy and paste the output to the dev.to UI</li>
Fill in the meta-date</li>
Hit publish</li>
</ol>
An ideal process would be:</p>

Write an Rmarkdown</li>
Hit publish</li>
</ol>
Lets see, that’s…</p>
removed_work <-</span> 4</span>/</span>6</span>

scales::</span>label_percent</span></span>(</span>)</span></span>(</span>removed_work)</span>
</span></code></pre>

  ## [1] "67%"
</span></code></pre>

</div>
Wow, gains</p>
Minimum Viable Function</h2>
So, the key piece of the package is to get a function that will jump my
markdown output from my computer onto dev.to. There are lots more bits
that I should have, but that part is the most important.</p>
#'</span> @title</span> Post a markdown file to dev.to
</span>#'</span> @description</span> Create a new post well rendered markdown file
</span>#'</span> @param</span> key</span> Your API key, Default: NA
</span>#'</span> @param</span> file</span> The path to the file, Default: file
</span>#'</span> @return</span> The response
</span>#'</span> @details</span> Will look for an api key in the `.REnviron` file. Seems to check if the body is identical to a previous article and error if so with `"Body markdown has already been taken"`.
</span>#'</span> @examples</span>
</span>#'</span> \dontrun{
</span>#'</span> if(interactive()){
</span>#'</span>  post_new_article("./articles/my_article.md")
</span>#'</span>  }
</span>#'</span> }
</span>#'</span> @seealso</span>
</span>#'</span>  \code{\link[httr]{POST}},\code{\link[httr]{add_headers}},\code{\link[httr]{verbose}},\code{\link[httr]{content}}
</span>#'</span>  \code{\link[readr]{read_file}}
</span>#'</span> @rdname</span> post_new_article
</span>#'</span> @export</span>
</span>#'</span> @importFrom</span> httr POST add_headers verbose content
</span>#'</span> @importFrom</span> readr read_file
</span>
post_new_article</span> </span><-</span> </span>function</span></span>(</span>key</span> =</span> NA</span>,</span> file</span> =</span> file)</span></span> {</span>

  response <-</span> httr::</span>POST</span></span>(</span>
    url</span> =</span> "</span>https://dev.to/api/articles"</span></span>,</span>
    httr::</span>add_headers</span></span>(</span>"</span>api-key"</span></span> =</span>
                        if</span> (</span>!</span>is.na</span></span>(</span>key)</span></span>)</span> {</span>
                          key
                        }</span> else</span> {</span>
                          message</span></span>(</span>"</span>Using DEVTO in .Renviron"</span></span>)</span></span>
                          Sys.getenv</span></span>(</span>x</span> =</span> "</span>DEVTO"</span></span>)</span></span>
                        }</span>)</span></span>,</span>
    body</span> =</span> list</span></span>(</span>article</span> =</span> list</span></span>(</span>
      title</span> =</span> '</span>title'</span></span>,</span>
      body_markdown</span> =</span> readr::</span>read_file</span></span>(</span>file</span> =</span> file)</span></span>
    )</span></span>)</span></span>,</span>
    encode</span> =</span> '</span>json'</span></span>,</span>
    httr::</span>verbose</span></span>(</span>)</span></span>
  )</span></span>
  httr::</span>content</span></span>(</span>response)</span></span>
}</span>
</span></code></pre>
Lots of similar things as the earlier function. Api key is all the same
code (don’t worry, when I have to write it a third time, I’ll abstract
it ;P). There are three key changes.</p>

use httr::POST</code> instead of httr::GET</code> because here I’m giving the
api info, not requesting it.</li>
use the body</code> argument to enclose the info I am sending the api,
with a list of 1 item article</code> which itself is a list of 2 items
title</code> and body_markdown</code></li>
use readr::read_file</code> to read the markdown file I want to post into
memory so it can be sent to the api</li>
</ol>
So does it work?</h2>
If you can read this, yes 🦾</p>


Tidy Tuesday and space to learn
2020-05-19T00:00:00+00:00
TidyTusdays</a> are a
weekly R Community event where people learn about RStats by practising
with a different data set each week. Last week the Cardiff R User
group</a> worked on the
volcanoes data-set :volcano:, and something in it really tripped me up.
We’ve done this a few times now, and are building up our work in this
GitHub repo</a>.</p>
library</span></span>(</span>tidyverse)</span></span>
volcano <-</span> readr::</span>read_csv</span></span>(</span>'</span>https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2020/2020-05-12/volcano.csv'</span></span>)</span></span>
</span></code></pre>
Data structure</h2>
Let’s have a look at what the data is</p>
volcano %>%</span> 
  str</span></span>(</span>)</span></span>
</span></code></pre>

  ## tibble [958 × 26] (S3: spec_tbl_df/tbl_df/tbl/data.frame)
##  $ volcano_number          : num [1:958] 283001 355096 342080 213004 321040 ...
##  $ volcano_name            : chr [1:958] "Abu" "Acamarachi" "Acatenango" "Acigol-Nevsehir" ...
##  $ primary_volcano_type    : chr [1:958] "Shield(s)" "Stratovolcano" "Stratovolcano(es)" "Caldera" ...
##  $ last_eruption_year      : chr [1:958] "-6850" "Unknown" "1972" "-2080" ...
##  $ country                 : chr [1:958] "Japan" "Chile" "Guatemala" "Turkey" ...
##  $ region                  : chr [1:958] "Japan, Taiwan, Marianas" "South America" "México and Central America" "Mediterranean and Western Asia" ...
##  $ subregion               : chr [1:958] "Honshu" "Northern Chile, Bolivia and Argentina" "Guatemala" "Turkey" ...
##  $ latitude                : num [1:958] 34.5 -23.3 14.5 38.5 46.2 ...
##  $ longitude               : num [1:958] 131.6 -67.6 -90.9 34.6 -121.5 ...
##  $ elevation               : num [1:958] 641 6023 3976 1683 3742 ...
##  $ tectonic_settings       : chr [1:958] "Subduction zone / Continental crust (>25 km)" "Subduction zone / Continental crust (>25 km)" "Subduction zone / Continental crust (>25 km)" "Intraplate / Continental crust (>25 km)" ...
##  $ evidence_category       : chr [1:958] "Eruption Dated" "Evidence Credible" "Eruption Observed" "Eruption Dated" ...
##  $ major_rock_1            : chr [1:958] "Andesite / Basaltic Andesite" "Dacite" "Andesite / Basaltic Andesite" "Rhyolite" ...
##  $ major_rock_2            : chr [1:958] "Basalt / Picro-Basalt" "Andesite / Basaltic Andesite" "Dacite" "Dacite" ...
##  $ major_rock_3            : chr [1:958] "Dacite" " " " " "Basalt / Picro-Basalt" ...
##  $ major_rock_4            : chr [1:958] " " " " " " "Andesite / Basaltic Andesite" ...
##  $ major_rock_5            : chr [1:958] " " " " " " " " ...
##  $ minor_rock_1            : chr [1:958] " " " " "Basalt / Picro-Basalt" " " ...
##  $ minor_rock_2            : chr [1:958] " " " " " " " " ...
##  $ minor_rock_3            : chr [1:958] " " " " " " " " ...
##  $ minor_rock_4            : chr [1:958] " " " " " " " " ...
##  $ minor_rock_5            : chr [1:958] " " " " " " " " ...
##  $ population_within_5_km  : num [1:958] 3597 0 4329 127863 0 ...
##  $ population_within_10_km : num [1:958] 9594 7 60730 127863 70 ...
##  $ population_within_30_km : num [1:958] 117805 294 1042836 218469 4019 ...
##  $ population_within_100_km: num [1:958] 4071152 9092 7634778 2253483 393303 ...
##  - attr(*, "spec")=
##   .. cols(
##   ..   volcano_number = col_double(),
##   ..   volcano_name = col_character(),
##   ..   primary_volcano_type = col_character(),
##   ..   last_eruption_year = col_character(),
##   ..   country = col_character(),
##   ..   region = col_character(),
##   ..   subregion = col_character(),
##   ..   latitude = col_double(),
##   ..   longitude = col_double(),
##   ..   elevation = col_double(),
##   ..   tectonic_settings = col_character(),
##   ..   evidence_category = col_character(),
##   ..   major_rock_1 = col_character(),
##   ..   major_rock_2 = col_character(),
##   ..   major_rock_3 = col_character(),
##   ..   major_rock_4 = col_character(),
##   ..   major_rock_5 = col_character(),
##   ..   minor_rock_1 = col_character(),
##   ..   minor_rock_2 = col_character(),
##   ..   minor_rock_3 = col_character(),
##   ..   minor_rock_4 = col_character(),
##   ..   minor_rock_5 = col_character(),
##   ..   population_within_5_km = col_double(),
##   ..   population_within_10_km = col_double(),
##   ..   population_within_30_km = col_double(),
##   ..   population_within_100_km = col_double()
##   .. )
</span></code></pre>

</div>
Objective</h2>
Lots of character columns, and some with some slightly funky formatting,
such as /</code> and variations on a theme with (s)</code> and (es)</code>. We’ve also
got a bunch of ‘sparse’ data in the columns that start with major_rock</code>
or minor_rock</code> that look like spaces. R has a rich set of tools for
dealing with missing data a little more effectively, so lets
clean this up by setting the missing data to be explicit NA</code>. In this
case, as the column is a character type, we need to NA_character</code> to
fill it up.</p>
Failing solution</h2>
volcano %>%</span>
    mutate_at</span></span>(</span>
    .vars</span> =</span> vars</span></span>(</span>starts_with</span></span>(</span>c</span></span>(</span>"</span>major_rock"</span></span>,</span> "</span>minor_rock"</span></span>)</span></span>)</span></span>)</span></span>,</span>
    .funs</span> =</span> ~</span> case_when</span></span>(</span>
      . ==</span> "</span> "</span></span> ~</span> NA_character_</span>,</span>
      TRUE</span> ~</span> .
    )</span></span>
  )</span></span> %>%</span> 
  select</span></span>(</span>starts_with</span></span>(</span>c</span></span>(</span>"</span>major_rock"</span></span>,</span> "</span>minor_rock"</span></span>)</span></span>)</span></span>)</span></span> %>%</span> 
  head</span></span>(</span>)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
major_rock_1</th> major_rock_2</th> major_rock_3</th> major_rock_4</th> major_rock_5</th> minor_rock_1</th> minor_rock_2</th> minor_rock_3</th> minor_rock_4</th> minor_rock_5</th></tr></thead>

Andesite / Basaltic Andesite</td> Basalt / Picro-Basalt</td> Dacite</td> </td> </td> </td> </td> </td> </td> </td></tr>
Dacite</td> Andesite / Basaltic Andesite</td> </td> </td> </td> </td> </td> </td> </td> </td></tr>
Andesite / Basaltic Andesite</td> Dacite</td> </td> </td> </td> Basalt / Picro-Basalt</td> </td> </td> </td> </td></tr>
Rhyolite</td> Dacite</td> Basalt / Picro-Basalt</td> Andesite / Basaltic Andesite</td> </td> </td> </td> </td> </td> </td></tr>
Andesite / Basaltic Andesite</td> Basalt / Picro-Basalt</td> </td> </td> </td> Dacite</td> </td> </td> </td> </td></tr>
Andesite / Basaltic Andesite</td> </td> </td> </td> </td> Dacite</td> Basalt / Picro-Basalt</td> </td> </td> </td></tr>
</tbody></table>
Well, that doesn’t quite work. What I want is to have the blank spots
filled up with NA</code>. Is it my code? It’s not the most basic solution,
using some of the tidyeval concepts such as vars</code> and funs</code>. Lets make
it as simple as possible.</p>
volcano %>%</span> 
  filter</span></span>(</span>major_rock_5 ==</span> "</span> "</span></span>)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>

</tbody></table>
Odd. I can’t find any values that are just spaces, even though they are
printed out that way! I know they are there, I can see them! Luckily,
the point of these projects is to learn, and to learn from each other in
the group :school:.</p>
Problem</h2>
My buddy Heather</a> was I think a little
surprised when I demonstrated this, but within a few minutes she’d
worked it out. It’s encoded as a non_breaking space</em>. She linked this
blog in our chat about non-braking
spaces</a>
and offered us the cryptic solution:</p>
"\u00A0"
</span></code></pre>
The blog goes into detail about what this is but the tl;dr is: “It looks
like a space but it’s not and it’s designed that
way</a>”.</p>
So a quick tweak to the code and…</p>
Functional solution</h2>
volcano %>%</span>
    mutate_at</span></span>(</span>
    .vars</span> =</span> vars</span></span>(</span>starts_with</span></span>(</span>c</span></span>(</span>"</span>major_rock"</span></span>,</span> "</span>minor_rock"</span></span>)</span></span>)</span></span>)</span></span>,</span>
    .funs</span> =</span> ~</span> case_when</span></span>(</span>
      . ==</span> "</span>\u</span>00A0"</span></span> ~</span> NA_character_</span>,</span>
      TRUE</span> ~</span> .
    )</span></span>
  )</span></span> %>%</span> 
  select</span></span>(</span>starts_with</span></span>(</span>c</span></span>(</span>"</span>major_rock"</span></span>,</span> "</span>minor_rock"</span></span>)</span></span>)</span></span>)</span></span> %>%</span> 
  head</span></span>(</span>)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
volcano_number</th> volcano_name</th> primary_volcano_type</th> last_eruption_year</th> country</th> region</th> subregion</th> latitude</th> longitude</th> elevation</th> tectonic_settings</th> evidence_category</th> major_rock_1</th> major_rock_2</th> major_rock_3</th> major_rock_4</th> major_rock_5</th> minor_rock_1</th> minor_rock_2</th> minor_rock_3</th> minor_rock_4</th> minor_rock_5</th> population_within_5_km</th> population_within_10_km</th> population_within_30_km</th> population_within_100_km</th></tr></thead>
major_rock_1</th> major_rock_2</th> major_rock_3</th> major_rock_4</th> major_rock_5</th> minor_rock_1</th> minor_rock_2</th> minor_rock_3</th> minor_rock_4</th> minor_rock_5</th></tr></thead>

Andesite / Basaltic Andesite</td> Basalt / Picro-Basalt</td> Dacite</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td></tr>
Dacite</td> Andesite / Basaltic Andesite</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td></tr>
Andesite / Basaltic Andesite</td> Dacite</td> NA</td> NA</td> NA</td> Basalt / Picro-Basalt</td> NA</td> NA</td> NA</td> NA</td></tr>
Rhyolite</td> Dacite</td> Basalt / Picro-Basalt</td> Andesite / Basaltic Andesite</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td></tr>
Andesite / Basaltic Andesite</td> Basalt / Picro-Basalt</td> NA</td> NA</td> NA</td> Dacite</td> NA</td> NA</td> NA</td> NA</td></tr>
Andesite / Basaltic Andesite</td> NA</td> NA</td> NA</td> NA</td> Dacite</td> Basalt / Picro-Basalt</td> NA</td> NA</td> NA</td></tr>
</tbody></table>
Full solution</h2>
Success! After digging around a little, I discovered str_trim</code> and
str_squish</code> can be used for this as well to make a perfectly tidy
solution!</p>
volcano %>%</span>
    mutate_at</span></span>(</span>
    .vars</span> =</span> vars</span></span>(</span>starts_with</span></span>(</span>c</span></span>(</span>"</span>major_rock"</span></span>,</span> "</span>minor_rock"</span></span>)</span></span>)</span></span>)</span></span>,</span>
    .funs</span> =</span> ~</span> case_when</span></span>(</span>
      str_trim</span></span>(</span>.)</span></span> ==</span> "</span>"</span></span> ~</span> NA_character_</span>,</span>
      TRUE</span> ~</span> .
    )</span></span>
  )</span></span> %>%</span> 
  select</span></span>(</span>starts_with</span></span>(</span>c</span></span>(</span>"</span>major_rock"</span></span>,</span> "</span>minor_rock"</span></span>)</span></span>)</span></span>)</span></span> %>%</span> 
  head</span></span>(</span>)</span></span> %>%</span> 
  knitr::</span>kable</span></span>(</span>)</span></span>
</span></code></pre>
major_rock_1</th> major_rock_2</th> major_rock_3</th> major_rock_4</th> major_rock_5</th> minor_rock_1</th> minor_rock_2</th> minor_rock_3</th> minor_rock_4</th> minor_rock_5</th></tr></thead>

Andesite / Basaltic Andesite</td> Basalt / Picro-Basalt</td> Dacite</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td></tr>
Dacite</td> Andesite / Basaltic Andesite</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td></tr>
Andesite / Basaltic Andesite</td> Dacite</td> NA</td> NA</td> NA</td> Basalt / Picro-Basalt</td> NA</td> NA</td> NA</td> NA</td></tr>
Rhyolite</td> Dacite</td> Basalt / Picro-Basalt</td> Andesite / Basaltic Andesite</td> NA</td> NA</td> NA</td> NA</td> NA</td> NA</td></tr>
Andesite / Basaltic Andesite</td> Basalt / Picro-Basalt</td> NA</td> NA</td> NA</td> Dacite</td> NA</td> NA</td> NA</td> NA</td></tr>
Andesite / Basaltic Andesite</td> NA</td> NA</td> NA</td> NA</td> Dacite</td> Basalt / Picro-Basalt</td> NA</td> NA</td> NA</td></tr>
</tbody></table>
Let’s try and breakdown what is happening in this solution by concept,
and then outline the routine in human language to finish.</p>
Concepts</h2>
Non-breaking spaces</h3>

the ‘missing’ values are not real spaces, they are non-breaking
spaces</em>.</li>
stringr::str_trim</code></a>
and stringr::str_squish</code> removes space from either the ends or all
the way through a character string depending on what else is
happening in the string and what you need from the solution.</li>
mutate_at</code></a>
is a buddy of mutate</code>, where you use functional programming</em> style
to apply a function over a collection of columns.</li>
this means that for the context of evaluation, we will be getting
""</code>, where as previously we were seeing " "</code> which was actually
encoded as "\u00A0"</code></li>
</ul>
Variable selection</h3>

starts_with</code></a>
is a select helper that is designed for cases when a related value
is stretch wide across multiple columns with similar names, and
returns a vector of column names filtered to your criteria.</li>
vars</code></a>
automatically quotes</em> the names of the columns to evaluate</em> later
in context and is almost always used as a wrapper to the .var =</code>
argument when it’s supported by a function.</li>
this means that we will be doing on operation on each of the columns
selected.</li>
</ul>
Formula function</h3>

.funs =</code> argument, like .var =</code>, has a counterpart
funs()</code></a>, but
this is being deprecated in favour of the expression notation</em>.</li>
case_when</code></a>
is an alternative version of the more common if else</code> operation.</li>
~</code> is a special operator that is key to the expression notation. It
effectively separates the Left Hand Side (LHS) of an expression from
the Right Hand Side (RHS). It’s used in two ways in this code.
case_when</code> uses full expressions to represent what should happen on
the RHS when the criteria of the LHS is met. .funs =</code> uses it to
make a lambda style formula. This is sometimes referred to as a
quosure</em></a>.</li>
.</code> is also a special operator, and I recommend reading the
documentation of
pipe</a> %>%</code>. The
idea of it here is to reference the data being operated on itself.
In this specific case, it’s each value from each of the selected
columns for equivalence to an empty space.</li>
</ul>
Step-by-step</h2>
Did you follow all that? It’s a minefield I know, but it allows us to do
something very powerful in only a few lines. As an alternative way of
understanding what this does, here’s the step by step:</p>

Get all the columns that start with either "major_rock'</code> or
"minor_rock"</code></li>
For each of the those columns trim any value at the start or end
that is whitespace, including non-breaking white space temporarily,
without modifying the underlying data</li>
If the value after that is an empty string, replace it in the same
column with the NA</code> value for characters</li>
If the value after does not pass that test, use the original value</li>
</ol>
So do you have to</em> program this way? No, not really. You could manually
create the list of columns you want to modify, but that would be prone
to human error and what if you end up with "minor_rock_6"</code> or
"major_rock_100"</code>? You could always make a traditional if else</code>
structure, but that can get long fast if there are multiple conditions
to check for. How about using the character string "\u00A0"</code> to test
for equivalence? Well, that would work now, but how do you pick up if
they change suddenly to actual spaces? Or another whitespace encoding?
Programming like this keeps code readable, maintainable, and robust.</p>
Is this overkill for tidy Tuesdays? Yes, absolutely. Are the problems
that you solve with this approach purely academic? No, not at all. Plus,
doing all that work in 129 characters is pretty neat. Excluding spaces.</p>


Pocket Monster BMI
2020-05-13T00:00:00+00:00
library</span></span>(</span>pokedex)</span></span>
library</span></span>(</span>tidyverse)</span></span>
library</span></span>(</span>knitr)</span></span>
</span></code></pre>
How big is a Pocket Monster?</h1>
Pokemon is a combination of ‘Pocket’ and ‘Monster’. So they’re all
pretty small right? Not quite.</p>
scale_format <-</span> scales::</span>number_format</span></span>(</span>accuracy</span> =</span> 1</span>,</span> big.mark</span> =</span> "</span>,"</span></span>)</span></span>

pokemon %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> height,</span> y</span> =</span> weight)</span></span>)</span></span> +</span>
  geom_point</span></span>(</span>)</span></span> +</span>
  scale_y_continuous</span></span>(</span>labels</span> =</span> scale_format)</span></span> +</span> 
  labs</span></span>(</span>title</span> =</span> "</span>Height and Weight of Pocket Monsters"</span></span>,</span>
       x</span> =</span> "</span>Height (m)"</span></span>,</span>
       y</span> =</span> "</span>Weight (kg)"</span></span>)</span></span>
</span></code></pre>
</p>
Maybe we need to work on this graph a little. Let’s make it log scaled on
both axis. I’ll put myself in for reference too.</p>
pokemon %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> height,</span> y</span> =</span> weight)</span></span>)</span></span> +</span>
  geom_point</span></span>(</span>)</span></span> +</span>
  geom_vline</span></span>(</span>xintercept</span> =</span> 1.</span>7</span>)</span></span> +</span>
  geom_hline</span></span>(</span>yintercept</span> =</span> 72</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Height and Weight of Pocket Monsters"</span></span>,</span>
       subtitle</span> =</span> "</span>Trainer Daves's height and weight for reference"</span></span>,</span>
       x</span> =</span> "</span>Height (m)"</span></span>,</span>
       y</span> =</span> "</span>Weight (kg)"</span></span>,</span>
       caption</span> =</span> "</span>Log X and Y Scale"</span></span>)</span></span> +</span>
  scale_x_log10</span></span>(</span>)</span></span> +</span>
  scale_y_log10</span></span>(</span>labels</span> =</span> scale_format)</span></span>
</span></code></pre>
</p>
So after a log transform of the scales we have a (roughly) linear
relationship. This is what we would expect in real world data. Nothing
with a physical existence can have a 0 or negative measure for these
features. Therefore, thinking of this in terms of average can be
misleading. It will always be ‘long tailed’. It might be a stretch to
refer to them all as Pocket Monsters though.</p>
There’s also clearly a relationship between both height and weight.
Let’s try and capture this in a single feature.</p>
BMI</h1>
The body mass index is a simple metric to link height and weight. Let’s
create it for our Pokemon.</p>
BMI</span> </span><-</span> </span>function</span></span>(</span>weight</span>,</span> height</span>)</span></span> {</span>
  weight/</span>(</span>height^</span>2</span>)</span>
}</span>

pokemon %>%</span> 
  mutate</span></span>(</span>BMI</span> =</span> BMI</span></span>(</span>weight</span> =</span> weight,</span> height</span> =</span> height)</span></span>)</span></span> -></span> pokemon

pokemon %>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> BMI)</span></span>)</span></span> +</span>
  geom_histogram</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>BMI of Pocket Monsters"</span></span>)</span></span>
</span></code></pre>
</p>
Well, that’s something of a surprise. Looking at the scatter plots from
earlier, there is something in the data set that is very small, but also
extremely</em> heavy. Let’s try and work out what that is.</p>
pokemon %>%</span> 
  arrange</span></span>(</span>desc</span></span>(</span>BMI)</span></span>)</span></span> %>%</span> 
  select</span></span>(</span>name,</span> height,</span> weight,</span> BMI,</span> genus)</span></span> %>%</span> 
  top_n</span></span>(</span>10</span>,</span> BMI)</span></span> %>%</span> 
  kable</span></span>(</span>)</span></span>
</span></code></pre>
name</th> height</th> weight</th> BMI</th> genus</th></tr></thead>

Cosmoem</td> 0.1</td> 999.9</td> 99990.0000</td> Protostar Pokémon</td></tr>
Minior</td> 0.3</td> 40.0</td> 444.4444</td> Meteor Pokémon</td></tr>
Aron</td> 0.4</td> 60.0</td> 375.0000</td> Iron Armor Pokémon</td></tr>
Durant</td> 0.3</td> 33.0</td> 366.6667</td> Iron Ant Pokémon</td></tr>
Clamperl</td> 0.4</td> 52.5</td> 328.1250</td> Bivalve Pokémon</td></tr>
Torkoal</td> 0.5</td> 80.4</td> 321.6000</td> Coal Pokémon</td></tr>
Cacnea</td> 0.4</td> 51.3</td> 320.6250</td> Cactus Pokémon</td></tr>
Munchlax</td> 0.6</td> 105.0</td> 291.6667</td> Big Eater Pokémon</td></tr>
Sandygast</td> 0.5</td> 70.0</td> 280.0000</td> Sand Heap Pokémon</td></tr>
Beldum</td> 0.6</td> 95.2</td> 264.4444</td> Iron Ball Pokémon</td></tr>
</tbody></table>
There’s no accounting for cosmological battle entities. I’m going to
claim that the Protostar Pokemon is a little out of scope for this and
filter it out. Let’s have a look at what we’re left with.</p>
Dave_BMI <-</span> BMI</span></span>(</span>weight</span> =</span> 72</span>,</span>  height</span> =</span> 1.</span>70</span>)</span></span>

pokemon %>%</span> 
  filter</span></span>(</span>name !=</span> "</span>Cosmoem"</span></span>)</span></span>%>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> BMI)</span></span>)</span></span> +</span>
  geom_histogram</span></span>(</span>)</span></span> +</span>
  geom_vline</span></span>(</span>xintercept</span> =</span> Dave_BMI)</span></span> +</span> 
  labs</span></span>(</span>title</span> =</span> "</span>BMI of Pocket Monsters"</span></span>,</span>
       subtitle</span> =</span> "</span>Trainer Dave's BMI for reference"</span></span>,</span>
       caption</span> =</span> "</span>Cosmoem removed"</span></span>)</span></span>
</span></code></pre>
</p>
So most Pokemon are actually a little bigger than me, and a few of them
are a lot bigger!
We’ve also realised that some of them might just be very different to
me, like stars, made entirely of metal or rock, or maybe even giant
dragons? Just like in the earlier article, I’m going to pivot the data
so we get a comparison for dual type Pokemon on both their types.</p>
pokemon %>%</span>
  filter</span></span>(</span>name !=</span> "</span>Cosmoem"</span></span>)</span></span> %>%</span> 
  select</span></span>(</span>name,</span> type_1,</span> type_2,</span> BMI,</span> height,</span> weight)</span></span> %>%</span>
  pivot_longer</span></span>(</span>
    cols</span> =</span> starts_with</span></span>(</span>"</span>type"</span></span>)</span></span>,</span>
    names_to</span> =</span> "</span>slot"</span></span>,</span>
    values_to</span> =</span> "</span>type"</span></span>,</span>
    values_drop_na</span> =</span> TRUE</span>
  )</span></span> %>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> BMI)</span></span>)</span></span> +</span>
  geom_density</span></span>(</span>)</span></span> +</span> 
  geom_vline</span></span>(</span>xintercept</span> =</span> Dave_BMI)</span></span> +</span> 
  facet_wrap</span></span>(</span>. ~</span> type,</span> scales</span> =</span> "</span>free"</span></span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Pocket Monster BMI by type"</span></span>,</span>
       subtitle</span> =</span> "</span>Trainer Dave's BMI for reference"</span></span>,</span>
       caption</span> =</span> "</span>Cosmoem removed"</span></span>)</span></span>
</span></code></pre>
</p>
So it looks like I’m not quite as hefty as Pokemon that are rock, steel,
ice, ground, fighting, dark or dragon. That makes sense. I’m also a bit
more corporeal than fairy or ghost type. Pokemon has some really big
bugs though!</p>
Game Over</h1>
We’ve learned quite a few R programming things today. The most obvious
were some ggplot</code> chart tools:</p>

geom_vline()</code> and geom_hline()</code> make 1 dimensional lines at
specific points</li>
geom_histogram()</code> and geom_density()</code> show the distribution of a
single value across mutiple observations</li>
facet_wrap()</code> can make grouped charts, which are often known as
small multiples</em></li>
scale_x_log10()</code> and scale_y_log10</code> is an easy way to plot a log
axis</li>
the scales</code> package is also useful for formatting the axis labels</li>
</ul>
Did you also notice the first thing we did with the scales</code> package? In
R you can assign a function</em> to a reference. This means that we don’t
need to repeat ourselves if we want to set it up with the same arguments
multiple times, like with formatting axis with large numbers.</p>
In the tidyverse</code> world we also used the optional arguments in
pivot_longer()</code> to select 2 columns to pivot on, and to drop rows we
create that have NA</code> when the Pokemon only has 1 type.</p>
Most importantly though, we created our own function, and it was easy!
The BMI</code> function we created we used to make a single value,
Dave_BMI</code>, but also to make the whole BMI</code> column for each Pokemon in
the data set! That’s pretty cool.</p>
From 2 known features, weight</code> and height</code> we made one single new
measurement BMI</code>. This an example of something that will come up more
in later posts about machine learning which is called ‘feature
engineering’.</p>
The next article will be going into how the Pokedex package is actually
made, both in trying to design a ‘tidy’ data set, but also how to make a
package in R!</p>
P.S. I know that Pocket Monster</code> is related to the pokeballs they fit in, but that’s a less fun title.</p>


Introducing the Pokedex package
2020-05-09T00:00:00+00:00
library</span></span>(</span>pokedex)</span></span>
library</span></span>(</span>tidyverse)</span></span>
library</span></span>(</span>knitr)</span></span>
</span></code></pre>
How many Pokemon in the package?</h2>
I’ve tried to make the data set ‘tidy’ from the start, so we can use
summarise</code> to count them, and kable</code> to make some dev.to friendly
markdown tables.</p>
pokemon %>%</span> 
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span> 
  kable</span></span>(</span>)</span></span>
</span></code></pre>
count</th></tr></thead>

807</td></tr>
</tbody></table>
Types</h2>
Types are pretty key to Pokemon. Lets have a quick look at the Kanto
starters and types.</p>
pokemon %>%</span> 
  top_n</span></span>(</span>n</span> =</span> -</span>9</span>,</span> wt</span> =</span> species_id)</span></span> %>%</span> 
  select</span></span>(</span>identifier,</span> type_1,</span> type_2)</span></span> %>%</span>
  kable</span></span>(</span>)</span></span>
</span></code></pre>
identifier</th> type_1</th> type_2</th></tr></thead>

bulbasaur</td> grass</td> poison</td></tr>
ivysaur</td> grass</td> poison</td></tr>
venusaur</td> grass</td> poison</td></tr>
charmander</td> fire</td> NA</td></tr>
charmeleon</td> fire</td> NA</td></tr>
charizard</td> fire</td> flying</td></tr>
squirtle</td> water</td> NA</td></tr>
wartortle</td> water</td> NA</td></tr>
blastoise</td> water</td> NA</td></tr>
</tbody></table>
Single and dual types</h2>
So Pokemon can have either 1 or 2 types. What’s the split between single
type and dual type Pokemon?</p>
pokemon %>%</span>
  mutate</span></span>(</span>dual_type</span> =</span> case_when</span></span>(</span>is.na</span></span>(</span>type_2)</span></span> ~</span> TRUE</span>,</span>
                               TRUE</span> ~</span> FALSE</span>)</span></span>)</span></span> %>%</span>
  group_by</span></span>(</span>dual_type)</span></span> %>%</span>
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span> 
  kable</span></span>(</span>)</span></span>
</span></code></pre>
dual_type</th> count</th></tr></thead>

FALSE</td> 405</td></tr>
TRUE</td> 402</td></tr>
</tbody></table>
So, it’s nearly a 50:50 split of Pokemon that are single type to Pokemon
that have 2 types.</p>
How many by type?</h2>
But there are also quite a few types of Pokemon. Starting with the
primary type, lets make a quick chart to understand the distribution of
primary types. Using group_by</code> will mean the summarise</code> gets
calculated per group</em>. We can then pipe directly into ggplot</code> for a
col chart with geom_col</code>.</p>
pokemon %>%</span>
  group_by</span></span>(</span>type_1)</span></span> %>%</span>
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> type_1,</span> y</span> =</span> count)</span></span>)</span></span> +</span>
  geom_col</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Pokemon by primary type"</span></span>)</span></span>
</span></code></pre>
</p>
Lots of water type Pokemon, and lots of normal type Pokemon, but very
few flying types. Interesting. How about the secondary types?</p>
pokemon %>%</span> 
  filter</span></span>(</span>!</span>is.na</span></span>(</span>type_2)</span></span>)</span></span> %>%</span> 
  group_by</span></span>(</span>type_2)</span></span> %>%</span> 
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> type_2,</span> y</span> =</span> count)</span></span>)</span></span> +</span>
  geom_col</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Pokemon by secondary type"</span></span>,</span>
       caption</span> =</span> "</span>For Pokemon with dual type"</span></span>)</span></span>
</span></code></pre>
</p>
Look at all those ’mons with flying as a secondary type! The thing is
that, game-wise, the order</em> of the typing doesn’t matter. We can easily
count the occurrence of a specific type in either primary or secondary
position with pivot_longer</code>.</p>
pivot_longer</code> is actually a newer tidyverse function. It is
complemented with pivot_wider</code> and this pair are intended to eventually
replace spread</code> and gather</code>. By filtering out the NA</code> I remove any
observations of secondary types for Pokemon that don’t actually have
them.</p>
pokemon %>%</span> 
  select</span></span>(</span>identifier,</span> type_1,</span> type_2)</span></span> %>%</span> 
  pivot_longer</span></span>(</span>-</span>identifier,</span> names_to</span> =</span> "</span>slot"</span></span>,</span> values_to</span> =</span> "</span>type"</span></span>)</span></span> %>%</span> 
  group_by</span></span>(</span>type)</span></span> %>%</span> 
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span> 
  filter</span></span>(</span>!</span>is.na</span></span>(</span>type)</span></span>)</span></span> %>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> type,</span> y</span> =</span> count)</span></span>)</span></span> +</span>
  geom_col</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Pokemon by either type"</span></span>,</span>
       caption</span> =</span> "</span>This will count a dual type Pokemon twice,\
once for each type"</span></span>)</span></span>
</span></code></pre>
</p>
So is there any consistency in order at all?</p>
pokemon %>%</span>
  filter</span></span>(</span>(</span>type_1 ==</span> "</span>ghost"</span></span> &</span> type_2 ==</span> "</span>fire"</span></span>)</span> |</span>
           (</span>type_1 ==</span> "</span>fire"</span></span> &</span> type_2 ==</span> "</span>ghost"</span></span>)</span>)</span></span> %>%</span> 
  select</span></span>(</span>identifier,</span> type_1,</span> type_2)</span></span>
</span></code></pre>
## # A tibble: 4 x 3
##   identifier  type_1 type_2
##   <chr>       <chr>  <chr> 
## 1 litwick     ghost  fire  
## 2 lampent     ghost  fire  
## 3 chandelure  ghost  fire  
## 4 blacephalon fire   ghost
</span></code></pre>
It doesn’t look like it. a ghost fire</code> Pokemon and a fire ghost</code>
Pokemon both turn up. I’d like to see what the coincidence rate is of
each type in dual type Pokemon, so I need to get some ordering in. I can
use case_when</code> in mutate</code> to create a two new columns in the data. I
can make 2 in one call because mutate</code> supports multiple expressions</em>,
each of which names a column, and then operates conditionally on the
other 2 type columns. These new columns will:</p>

always have a value in the column type_1_ordered</code></li>
if the Pokemon is dual type, have a value in type_2_ordered</code></li>
always have the types alphabetically ordered between the two
columns. i.e. it will always be fire, ghost</code>, never ghost, fire</code>.</li>
</ul>
pokemon %>%</span>
  mutate</span></span>(</span>
    type_1_ordered</span> =</span> case_when</span></span>(</span>is.na</span></span>(</span>type_2)</span></span> ~</span> type_1,</span>
                               type_1 <</span> type_2 ~</span> type_1,</span>
                               TRUE</span> ~</span> type_2)</span></span>,</span>
    type_2_ordered</span> =</span> case_when</span></span>(</span>type_1 ></span> type_2 ~</span> type_1,</span>
                               TRUE</span> ~</span> type_2)</span></span>
  )</span></span> -></span> pokemon
</span></code></pre>
What might the distribution be of the flying secondary type, per primary
type?</p>
pokemon %>%</span>
  mutate</span></span>(</span>type_combined</span> =</span> case_when</span></span>(</span>
    !</span>is.na</span></span>(</span>type_2_ordered)</span></span> ~</span> paste</span></span>(</span>type_1_ordered,</span> type_2_ordered)</span></span>,</span>
    is.na</span></span>(</span>type_2_ordered)</span></span> ~</span> type_1_ordered
  )</span></span>)</span></span> %>%</span>
  group_by</span></span>(</span>type_combined)</span></span> %>%</span>
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span>
  arrange</span></span>(</span>desc</span></span>(</span>count)</span></span>)</span></span> %>%</span>
  mutate</span></span>(</span>type_combined</span> =</span> as_factor</span></span>(</span>type_combined)</span></span>)</span></span> %>%</span>
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> type_combined,</span> y</span> =</span> count)</span></span>)</span></span> +</span>
  geom_col</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Count of Pokemon by dual type"</span></span>,</span>
       caption</span> =</span> "</span>Ordered by count"</span></span>)</span></span> +</span> 
  theme</span></span>(</span>axis.text.x</span> =</span> element_text</span></span>(</span>angle</span> =</span> 90</span>)</span></span>)</span></span>
</span></code></pre>
</p>
So the most often occurring dual type is flying normal. That explains
the first 2 charts. It’s a bit tricky to see the rest though. Lets make
a more useful plot.</p>
pokemon %>%</span> 
  group_by</span></span>(</span>type_1_ordered,</span> type_2_ordered)</span></span> %>%</span> 
  filter</span></span>(</span>!</span>is.na</span></span>(</span>type_2_ordered)</span></span>)</span></span> %>%</span> 
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span> 
  ggplot</span></span>(</span>aes</span></span>(</span>x</span> =</span> type_1_ordered,</span> y</span> =</span> type_2_ordered,</span> size</span> =</span> count)</span></span>)</span></span> +</span>
  geom_point</span></span>(</span>)</span></span> +</span>
  labs</span></span>(</span>title</span> =</span> "</span>Coincidence of a particular dual type"</span></span>)</span></span>
</span></code></pre>
</p>
So flying normal</code> has the biggest count, with there being quite a few
bug flying</code>. That makes sense, as so many bug Pokemon have wings!
There are also lot’s of bug poison</code> and grass poison</code>. That makes
sense too, as so many bugs and plants are poisonous! How many Pokemon
have unique types though?</p>
pokemon %>%</span> 
  group_by</span></span>(</span>type_1_ordered,</span> type_2_ordered)</span></span> %>%</span> 
  filter</span></span>(</span>!</span>is.na</span></span>(</span>type_2_ordered)</span></span>)</span></span> %>%</span> 
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span> 
  filter</span></span>(</span>count ==</span> 1</span>)</span></span> %>%</span> 
  ungroup</span></span>(</span>)</span></span> %>%</span> 
  summarise</span></span>(</span>count</span> =</span> n</span></span>(</span>)</span></span>)</span></span> %>%</span> 
  kable</span></span>(</span>)</span></span>
</span></code></pre>
count</th></tr></thead>

24</td></tr>
</tbody></table>
24 Pokemon have unique dual types. Out of 807 that isn’t very many!
Maybe these Pokemon might be particularly useful? I’ll try and work it
out…</p>
Game Over</h2>
This post has been a simple example of both the data in the package, but
also the tidyverse</code> methods of doing Exploratory Data Analysis. You can
find out more about tidyverse here</a></p>
I got the raw data from this repo by
veekun</a>. My package is available
here</a>, and the particular version
I used for this post is
here</a>.
Though it’s in a pretty raw state, I hope to improve over time.</p>
I made this package to have a bigish, diverse set of data to play with,
that lots of people recognise, and that has some inherent real world
application. Pokemon is a huge franchise with multiple instalments. Lots
of people have played it, and even if you haven’t you probably have an
intuition about what a Pokemon is, and what data about a Pokemon might
make sense, and mean in context with other Pokemon. Feel free to fork
and mess around with as you like. I hope its fun, and maybe even
useful!</p>


Local gitlab runners, 'no such image', docker and disk space
2020-01-10T00:00:00+00:00

  Davids-MacBook-Pro:data-science davidparr$ gitlab-runner exec docker 'anomaly detection'
Runtime platform                                    arch=amd64 os=darwin pid=72331 revision=a8a019e0 version=12.3.0
WARNING: You most probably have uncommitted changes. 
WARNING: These changes will not be tested.         
Running with gitlab-runner 12.3.0 (a8a019e0)
Using Docker executor with image rocker/tidyverse:latest ...
Authenticating with credentials from /Users/davidparr/.docker/config.json
Pulling docker image rocker/tidyverse:latest ...
ERROR: Preparation failed: Error: No such image: rocker/tidyverse:latest (executor_docker.go:195:0s)
Will be retried in 3s ...
Using Docker executor with image rocker/tidyverse:latest ...
Authenticating with credentials from /Users/davidparr/.docker/config.json
Pulling docker image rocker/tidyverse:latest ...
ERROR: Preparation failed: Error: No such image: rocker/tidyverse:latest (executor_docker.go:195:0s)
Will be retried in 3s ...
Using Docker executor with image rocker/tidyverse:latest ...
Authenticating with credentials from /Users/davidparr/.docker/config.json
Pulling docker image rocker/tidyverse:latest ...
ERROR: Preparation failed: Error: No such image: rocker/tidyverse:latest (executor_docker.go:195:0s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error: No such image: rocker/tidyverse:latest (executor_docker.go:195:0s)
</span></code></pre>

</div>

But you know</em> the image exists. It’s definately</em> a thing. Look, it’s right here</a>. I can even RUN IT IN PRODUCTION</strong> so why is it not running on my machine?</p>
</p>
Turns out gitlab, as brilliant as I’m normally finding their CICD solution, is lying to you. The image does</em> exist, you aren’t</em> mad, just not seeing the whole picture. </p>

  Davids-MacBook-Pro:data-science davidparr$ docker pull rocker/tidyverse
Using default tag: latest
latest: Pulling from rocker/tidyverse
16ea0e8c8879: Pull complete 
7ce39da2c1e2: Extracting [==================================================\u003e]  222.8MB/222.8MB
ff1bceed0bef: Download complete 
e36d273bec5a: Download complete 
d3acc34c6c77: Download complete 
14d07989ce8b: Download complete 
73b6bcbfcb26: Download complete 
70b803ec0e47: Download complete 
failed to register layer: Error processing tar file(exit status 1): write /usr/lib/gcc/x86_64-linux-gnu/8/libsupc++.a: no space left on device
Davids-MacBook-Pro:data-science davidparr$ docker pull rocker/tidyverse
Using default tag: latest
latest: Pulling from rocker/tidyverse
16ea0e8c8879: Pull complete 
7ce39da2c1e2: Extracting [===================\u003e                               ]  85.23MB/222.8MB
ff1bceed0bef: Download complete 
e36d273bec5a: Download complete 
d3acc34c6c77: Download complete 
14d07989ce8b: Download complete 
73b6bcbfcb26: Download complete 
70b803ec0e47: Downloading [==================================================\u003e]  324.8MB/324.8MB
write /var/lib/docker/tmp/GetImageBlob686008714: no space left on device
</span></code></pre>

</div>
That’s</em> the error message we needed. It turns out that after a while, your local machines ‘Sparse Image’, that has all your docker images, containers, networks and registries, and also resizes it’s on disk footprint as you use docker, will get filled up with old images, containers and other cruft. </p>
There is a bit more info in this thread</a> but the short version is that you probably just want to run docker system prune -a</code>.</p>
This command is documented to "Remove unused data", and when run will tell you all about what it’s going to do:</p>
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache
</span></code></pre>
For my use cases this is perfect. All the images I need in practice are backed up on the cloud and can be rebuilt from code when needed (as they should be), leaving my disk memory to be able to be treated a little more like active memory, hot swapping in and out containers as needed, keeping the disk space tidy and minimal. </p>
In the end I received this lovely message:</p>
Total reclaimed space: 10.24GB
</span></code></pre>
And we’re back to happy local emulation of my CI/CD pipeline. Now if only I could optimise my R testing image…</p>


The Real Difference (TM) between Python and R for Data Science
2019-08-27T00:00:00+00:00
Batman vs Superman</h2>
Data science has a two language problem. R and python are both</strong> the</em> language for data science. This has lead to some pretty abstract, generic and sometimes absurd definitions between the two languages. The worst is probably this image:</p>
</p>
Data science is such a wide term, with sometimes a very diverse or even poor understanding in business, that clickbaity titles giving superficial answers abound. I’ve tracked down the source of this image</a>, which has also floated around on twitter once (2016)</a> or twice (2018)</a>. To give the author of the post his dues, the article it’s derived from is interesting, and I hope the SEO bonus from publishing as the film hype cycle grew got him some extra book sales (the books he links to are actually really</em> worth reading).</p>
I first came across it on (a now removed) instagram post where it was literally captioned "How to pass any data-science interview" on an account specifically advertising it’s data science recruitment services. I’m not against pop-culture/programming bleed over in any form (I’ve done enough Pokemon programming talks that any horse I get on has to be less Rapidash and more Ponyta), but at the point that people who are trying to break into the industry are being told this is valuable information to tell an interviewer</em>, then there is evidently a gap between needs and knowledge.</p>
The difference, and decision for the ‘right’ tool for the job is actually defined by a mixture of internal and external factors. Both languages, in themselves, do the job of data ingest, processing, analysis, modelling and prediction equally well. Both can be deployed on-premises or on the cloud. Both  Minor differences in syntax, and major differences in deployment tooling become the defining factors.</p>
INTERNALS: Syntax and ecosystem</h2>
Any programming language is little more than syntax. Functional, Object-oriented, Imperative and more, in various combinations, with specific conventions and patterns. Curiously both R</a> and Python</a> are ‘Multi-paradigm’ according to Wikipedia. This means that multiple, different styles can be used depending on the problem at hand. You can write python in an OO way, or a functional way. R is often thought of as functional, though it allows the construction of classes, and objects with methods, and side-effects, so it’s not a ‘pure functional’ language.</p>
However, how do people</em> actually write code for data science?</h3>
In R, data science is a first class application</em> (and maybe even sole aplication</em>?). Tidyverse</a> is arguably the de facto way to write most R projects now. With RStudio giving it full corporate support and funding an eco-system of tooling, most R I write, have seen others write, and I’ve taught people to write is in this style. Pipes -></code> chain function calls with non-standard evaluation of arguments, Rmarkdown documents, ggplot visualisation and shiny apps are an R data scientists solution of choice. Tidyverse is itself functionally inspired</em>, purrr explicitly states this</a> and Hadley Wickham directly argues that "R, at its heart, is a functional programming (FP) language."</a></p>
Python is everywhere</em>. Data science in Python isn’t pythons only reason for existence. scipy</a> is the python tool chain for data science. numpy and pandas are best of buddies, and matplotlib is a front runner for graphics. This collection of tools is far less cohesive though, with scikit learn, seaborn and jupyter notebooks also being hugely popular, but totally removed from the scipy ecosystem. These tools therefore need a bit more work to mould into a full product. Object Oriented programming is also more popular, even for</a> data science</a>. This means you’ll be calling objects more directly, and explicitly pulling out methods on objects with myObject.method</code> more often.</p>
What does that mean for me?</h3>
Do you understand functional programming or object oriented programming already, or is one immediately making sense over the other? Great, use that one. 
Do you already have an ecosystem of certain tools? Great, use whatever you already have.
Do you already have colleagues that you need to work with? Great, use whatever they are.
Do you already have a support network around you for one language but not the other? Great, don’t make life harder than it needs to be.</p>
EXTERNALS: Application and deployment</h2>
What if you don’t have an easy answer though? You’re starting a new project, or you are the only programmer in the business? Then the choice becomes harder to get right, but more clear the more experience you have. When you know both tools can achieve</em> the same job, you have to choose which one does it more easily</em>. I’ve found that I will reach for python and R equally now I’ve been working professionally in both for a while, however, which</em> I use is defined pretty clearly.</p>
When to use R</h3>
R encapsulates statistic and mathematical ideas clearly and robustly. Depending on where you get your dependencies from you can have a very clear idea of the likely ‘correctness’ of the library. Roughly ROpenSci</a> is more rigorous and stable than CRAN</a> which is better than github. Tidyverse pipes, NSE, purrr give a tightly coupled working environment where syntax is consistent, terse, and trivially refactored. ggplot and RMarkdown give business consumable outputs from the start, where code can be kept tightly coupled to narrative and reporting, but also infinitely customisable to produce something that would get past marketing without a glance. Many academic publications rely on R and RMarkdown for both research and publication. If you need to make interactive output, shiny is a straightforward application framework, with a tight to everything else in this ecosystem.</p>
R is for analysts</strong> who need to be certain</em> of the underlying data processing immediately</em>, produce and iterate on reporting</em> outputs as fast as possible</em> and write the least code</em> for the most return. Hadley effectively said as much last week</a>.</p>

I think R Markdown is an amazing contribution to R. … When you are doing data analysis typing speed is actually a bottleneck.</p>
</blockquote>
When to use Python</h3>
Python is already used tonnes more than R</a> because data science isn’t it’s only</em> job. Data engineers use it very heavily, back-end and even front-end developers use it for all kinds of projects, wether it is data intensive or not. In many linux distributions (including Mac) it’s actually installed as part of your machine when you put the operating system on. It’s also arguably a cloud native</em> language. AWS lambda functions support it out of the box</a>, and Microsoft recently copied them</a>. This is doubly interesting as Microsoft bought one of the biggest R consultancies in 2015</a>, but as of yet R is not a natively supported language in much of Microsofts ecosystem. It’s definitely got more baked in R support than AWS (PowerBI</a> and many others</a>), but has made moves to potentially trim back MRAN</a>. All this means that Python is spoken by most developers, that python is supported by most big cloud providers, and that python is probably already built into whatever you are working on.</p>
Python is for developers</strong> who need to deploy software</em> within a traditional software environment</em> in a more traditional development</em> workflow, where first-class cloud support</em> matters, and integration with existing code</em> is of top priority.</p>
Conclusion</h2>
R doesn’t necessarily deploy easily into software development</a> (not strictly related, but great talk that’s relevant to the problem), though it is definitely possible (I’ve done it), you have to put in more work to get a roughly similar result. In many situations, that trade off might not be worth it.</p>
Python doesn’t necessarily fit into BI and academic workflows, thought it’s definitely possible (I’ve also done it), you have to put in more work to get a roughly similar result. In many situations, that trade off might not be worth it.</p>
Superman, Batman, Detective Work, Intelligence, Cunning, Usage of Tools, More Brain than Muscles, Muscle Power, Super Strength, Elegance, Wide Range, More Muscles than Brain are not meaningful differentiators.</p>

Semantic vs keyword</h3>

Keyword</h4>
A keyword search looks for the exact letters in a string, or potentially a partial match. As an example the query `"Data Science"</code> would find things that exactly matched the charcters in the string "Data Science"</code> and maybe also ["Data", "Science", "DS"]</code>.</p>`

Click</h3>
pallets/click</a></p>
I actually am using `click</code>, sort of. typer</code> is built ontop of click</code>, but to be honest I didn’t really know that before I’d mostly decided. click</code> looks like a really great project, but it wasn’t as</em> clear how to get started.</p>`

What do you think?</h2>
Does this sound like something intersting to you, maybe even something useful? Did this just spark inspiration in you for a new project? Does this actually already exist somewhere and I’m just being an idiot? Let me know :)</p>

Background</h2>
I made it because I saw this repo by zhiiyang</p>
zhiiiyang/zhiiiyang</a></p>
Thanks to @mokkapps for this article which uses the twitter updating action for discovering it.</p>
How I built a self updating readme</a></p>

Solution</h1>
Nativefier</a></p>
This post</a> and this one</a> helped iron out some kinks and now I can launch programming music right from my VS code terminal!</p>
</p>

Extra credit</h1>
`alias</span> musicforprogramming</span>=</span>"</span>~/Dev/musicforprogramming-linux-x64/musicforprogramming </span></span></span></span></code></pre>`

`TL;DR</h1> Start your package in the RStudio IDE as a new project</li> Set up your package with usethis</code></li> Populate your Roxygen comments with sinew</code></li> Profit!</li> </ol>`

Hiccup</h2>
I was a little surprised when my local copy suddenly got all my blog files that I then just commited</a>, but that seemed to be ok. Pretty sure I’ve not broken anything :)</p>

P.S.</h2>
If you are reading this on</em> my website</a>, this is all actually managed through a headless CMS called dev.to</a>. You can use the link to my profile on the left of this article. If you are reading this on dev.to you can see my running website at daveparr.info</a>.</p>

Notable changes</h2>

So does it work?</h2>
If you can read this, yes 🦾</p>

Concepts</h2>

Pocket Monster BMI

Introducing the Pokedex package

Local gitlab runners, 'no such image', docker and disk space

The Real Difference (TM) between Python and R for Data Science

DaveParr.Info

Moving Starpilot to GraphQL

How I did it</h2>

Copilot for your GitHub stars

My GitHub profile shows my popular DEV.to posts and GitHub repos automatically

How to calculate a Pokemons 'power level' using kmeans

The Missingno Experiment and Multiple Form Pokemon

Why did I make this dev.to API wrapper?

Nativefier is bonkers

Building my first Django project with CSS and Static Files

Why use AWS Lambda for Data Science?

3 minimal features for my dev.to api wrapper

Webscraping with rvest and themeing ggplot

Writing R packages, fast

Building my first Django project with a Database

Building my first Django project with pages

Testing my dev.to API package with testthat, webmockr and vcr

Building my first Django project

Setting up a Django dev environment

I made my dev.to content into a website to find a new job

Posting straight from Rmd to DEV.TO (for real this time)

Posting from .Rmd to dev.to

Tidy Tuesday and space to learn

DaveParr.Info

Moving Starpilot to GraphQL

How I did it</h2>

Copilot for your GitHub stars

The original idea</h2> Use a dataset of your personal stars to inform retrival augmented generation for a question and answer large language model deployed in a command line interface</p> </blockquote> I thought this would be useful for a few reasons:</p>

Semantic vs keyword</h3>

Keyword</h4> A keyword search looks for the exact letters in a string, or potentially a partial match. As an example the query "Data Science"</code> would find things that exactly matched the charcters in the string "Data Science"</code> and maybe also ["Data", "Science", "DS"]</code>.</p>

Vectorstores</h2> A type of data base called a vectorstore is commonly used for this because they are deliberately optimised towards a similarity search use case. They achieve this in a few ways:</p>

Designing a system</h2> With this set of goals and new knowledge I got to work working out which puzzle pieces I needed and how to fit them together. This time I did go through my stars (and a few other things), though maybe this is for the last time!</p>

What do you think?</h2> Does this sound like something intersting to you, maybe even something useful? Did this just spark inspiration in you for a new project? Does this actually already exist somewhere and I’m just being an idiot? Let me know :)</p>

My GitHub profile shows my popular DEV.to posts and GitHub repos automatically

How to calculate a Pokemons 'power level' using kmeans

The Missingno Experiment and Multiple Form Pokemon

Why did I make this dev.to API wrapper?

Most of the R users on the site syndicate</h3> There are some really great R users on here already:</p>

Nativefier is bonkers

Motivation</h1> I like music, but don’t like music in browser tabs. Basically because i have it open all the time, I want to find and control it easily, and I don’t want it cluttering up an area that might be soley focused on work. </p>

Building my first Django project with CSS and Static Files

Why use AWS Lambda for Data Science?

3 minimal features for my dev.to api wrapper

Webscraping with rvest and themeing ggplot

Writing R packages, fast

TL;DR</h1> Start your package in the RStudio IDE as a new project</li> Set up your package with usethis</code></li> Populate your Roxygen comments with sinew</code></li> Profit!</li> </ol>

Building my first Django project with a Database

Building my first Django project with pages

Testing my dev.to API package with testthat, webmockr and vcr

Building my first Django project

Setting up a Django dev environment

I made my dev.to content into a website to find a new job

Posting straight from Rmd to DEV.TO (for real this time)

Notable changes</h2>

Posting from .Rmd to dev.to

So does it work?</h2> If you can read this, yes 🦾</p>

Tidy Tuesday and space to learn

Concepts</h2>

Keyword</h4>
A keyword search looks for the exact letters in a string, or potentially a partial match. As an example the query `"Data Science"</code> would find things that exactly matched the charcters in the string "Data Science"</code> and maybe also ["Data", "Science", "DS"]</code>.</p>`

What do you think?</h2>
Does this sound like something intersting to you, maybe even something useful? Did this just spark inspiration in you for a new project? Does this actually already exist somewhere and I’m just being an idiot? Let me know :)</p>

`TL;DR</h1> Start your package in the RStudio IDE as a new project</li> Set up your package with usethis</code></li> Populate your Roxygen comments with sinew</code></li> Profit!</li> </ol>`

So does it work?</h2>
If you can read this, yes 🦾</p>