Author Archives: Jeff

Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query

Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query

This is kind of a silly and duplicative post, but I spent too much time searching for the right answer, so maybe this will help the right course of action bubble to the top faster in the future.

The Issue

I was trying to run a query on my local SQL install (whatever MAMP manages and provisions) using MySQL Workbench 6.3 for Mac but kept getting a timeout error.

The query itself wasn’t overly complex, but I was using aggregate functions, group by, and a join to consolidate a dataset. I’m working with distance education reporting data for all U.S. colleges and universities from 2012-2015, so this join involved a 7K row table and another with 25K rows, so not inconsequential but also not BIG data level.

SELECT
STABBR as State,
EFDELEV as Level , 
SUM(EFDETOT) as Total_Distance,
SUM(EFDEEXC) as Exclusive_Distance,
SUM(EFDESOM) as Some_Distance,
SUM(EFDENON) as None_Distance

FROM hd2012 LEFT JOIN ef2012a_dist_rv
ON hd2012.UNITID = ef2012a_dist_rv.UNITID
GROUP BY State,  Level;

I did some initial googling on the error code, but it is a pretty general error code, so it was difficult to be sure whether this was a limitation of SQL or the Workbench DBMS. I read a few posts that suggested manipulating some of the .conf files for the underlying MySQL install, and I went too long down this road before trying something in Workbench itself.

It turns out there are timeout settings for the DBMS that you extend to make sure that it waits a sufficient amount of time for your query to return data. Thanks to this specific answer on StackOverflow, but the description of “how-to” it links to is no longer valid, hence this blog post.

The Fix

There is a quick setting in Preferences that helped me. As you might expect, the DBMS has settings to manage its connection to the SQL server. In my case, those were just too short for my long running queries.

I changed the 30 second defaults to 180, and returned the data I needed. However, I’d imagine that some things would call for a much higher timeout, especially if you wanted to do a lot of transactions.

Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query

 

 

 

 

The post Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query appeared first on Jeff Everhart.

Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query

Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query

This is kind of a silly and duplicative post, but I spent too much time searching for the right answer, so maybe this will help the right course of action bubble to the top faster in the future.

The Issue

I was trying to run a query on my local SQL install (whatever MAMP manages and provisions) using MySQL Workbench 6.3 for Mac but kept getting a timeout error.

The query itself wasn’t overly complex, but I was using aggregate functions, group by, and a join to consolidate a dataset. I’m working with distance education reporting data for all U.S. colleges and universities from 2012-2015, so this join involved a 7K row table and another with 25K rows, so not inconsequential but also not BIG data level.

SELECT
STABBR as State,
EFDELEV as Level , 
SUM(EFDETOT) as Total_Distance,
SUM(EFDEEXC) as Exclusive_Distance,
SUM(EFDESOM) as Some_Distance,
SUM(EFDENON) as None_Distance

FROM hd2012 LEFT JOIN ef2012a_dist_rv
ON hd2012.UNITID = ef2012a_dist_rv.UNITID
GROUP BY State,  Level;

I did some initial googling on the error code, but it is a pretty general error code, so it was difficult to be sure whether this was a limitation of SQL or the Workbench DBMS. I read a few posts that suggested manipulating some of the .conf files for the underlying MySQL install, and I went too long down this road before trying something in Workbench itself.

It turns out there are timeout settings for the DBMS that you extend to make sure that it waits a sufficient amount of time for your query to return data. Thanks to this specific answer on StackOverflow, but the description of “how-to” it links to is no longer valid, hence this blog post.

The Fix

There is a quick setting in Preferences that helped me. As you might expect, the DBMS has settings to manage its connection to the SQL server. In my case, those were just too short for my long running queries.

I changed the 30 second defaults to 180, and returned the data I needed. However, I’d imagine that some things would call for a much higher timeout, especially if you wanted to do a lot of transactions.

Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query

 

 

 

 

The post Query Timeout in MySQL Workbench | Error Code: 2013. Lost connection to MySQL server during query appeared first on Jeff Everhart.

Extending WP REST API Index Route Response

Extending WP REST API Index Route Response

This should be a fairly quick blog post, but it should help some folks out if they are looking to extend the WP Rest API index route to include some additional fields. First, let’s clear up what I mean by index route. 

For the purpose of this post, we are considering the index route to be whatever is returned when you request the /wp-json/ endpoint, not the fully qualified namespace /wp-json/wp/v2, which returns route information about the site. The /wp-json endpoint returns some of that information as well, but it also includes some details about the site itself. 

We’re prototyping some additional ways to aggregate student portfolio pages/posts through the API, and we need to eventually develop a plugin to add some additional site-level settings that will help to structure the various portfolio views. 

Either way, we needed to add some additional fields to the index response, but the WP docs aren’t that specific for this endpoint. Most of the existing resources on modifying responses out there deal with adding fields to existing objects like posts, pages, or comments. 

However, there isn’t an Index object that we can pass through, which lead us to some available filter options using the ‘rest_index’ filter. From there, we get access to the WP_REST_Response object and can modify it from there. As it turns out, most of the important stuff going on happens on the WP_HTTP_Response object, so that is a better place to start if you’re looking to modify the response object in a meaningful way.  

At the end of the day, the data on the response object is just an associative array, and you can modify it as you would any other 2D array. Here is the code that should go in functions.php: 

<?php

function filterResponse($response){
   $data = $response->data;
   $data['extra_field'] = 'some data';
   $response->set_data($data);
   return $response;
}

add_filter('rest_index', 'filterResponse');

?>

 

The post Extending WP REST API Index Route Response appeared first on Jeff Everhart.

Using AmCharts with Vue and Webpack

Using AmCharts with Vue and Webpack

I finally swallowed the Webpack pill, mostly because I wanted to get the most out of single file Vue components for some new projects I’m working on, and Webpack is along for the ride.

Overall, it’s been a semi-frustrating but also instructive experience. Before this I never used ESLint or any other type of linting, and I’m still pretty on the fence about its usefulness outside of really large and complex projects, but it’s taught me a few things about what people consider ‘modern’ JavaScript to look like.

Webpack itself seems to be like Gulp on steroids and a bit more to wrap your head around. However, the biggest change it necessitates for me is adoption of ES6 module patterns. I’m familiar with this type of pattern from my work on Node projects, but its usage with single file Vue components was new to me.

More importantly, a lot of the previous patterns I’ve used to build things on the front end are now pretty much bunk. I’ll talk about a few ways to work around some module conventions if you are using a project that doesn’t support that method.

The Issue at Hand

In a lot of my previous projects, I handled dependency management in a more straightforward fashion. If the project was small, a few script tags in the body sufficed. If it was larger, I would use Gulp to minify and concatenate all of the files together into one bundle.

Webpack, on the hand, introduces the idea of the dependency graph and organizes your code so that each module is self-contained, and only has access to the code it needs to do its job. Overall, this promotes good design patterns and reduces the amount of bugs that can be introduced into your code.

While that’s all well and good, those benefits come at the cost of added complexity, especially if the code you use to build things doesn’t adhere to the module pattern. In this case, Amcharts is a library I use quite frequently for charting, and it wasn’t immediately compatible with Vue and Webpack.

The fix was fairly easy, but I figured I’d write a blog post to help further my own understanding as well as add to the searchable material that others might find useful.

Understanding the Module Pattern

To better understand how to make random things work with Webpack, let’s look at how it works and what it needs/expect to do what it does. Before advanced JS bundling, most library developers followed the practices outlined below:

window.sweetLibrary = {
      method: function () {
       console.log('sweet method')
      }, 
      property: 'sweet prop'
}


sweetLibaray.method()

You would define a global variable or attach an object to the global window object, which are both ways of saying make the library interface available to any JavaScript program executing in the same thread.

This is pretty straightforward, but not without its downsides. If we have a bunch of different libraries mucking around in the global scope, the potential for namespace collisions goes up. Lots of smart developers came up with ways to make this more sensible for larger-scale projects, mostly involving closures, but I won’t get into all that. There is a great article from the Free Code Camp community that sums up JS modules better than I could.

So, how is the module pattern different?

Instead of making our JS libraries available to the global scope, we can explicitly export objects and import them only when needed.

For example, we might define a locally scoped object, and then expose that through a module interface using module.exports:

//sweet-library.js

let sweetLibrary = {
    method: function () {
      console.log('sweet library')
    }, 
    prop: 'sweet prop'
}

module.exports = sweetLibrary

Here we are exporting an object, but we can really export anything we want. Once we’ve exported something, we can import or require that module in another piece of code to gain access to its functionality.

//sweet-component.js 
let sweetLibrary = require('sweet-library')
sweetLibrary.method()
//sweet method

//Or, we could use import 
import sweetLibrary from 'sweet-library'
sweetLibrary.method()
//sweet method

Mostly these two methods do the same thing by giving you access to another library or object through this export/import pattern. And, more importantly, this is the way the Webpack expects you to use JS code if you want it to be managed by the Webpack build process.

Getting Amcharts to Work (using non-module libraries with Webpack)

Back to my initial issue. I have this library I use a lot for charting, and want to use it within my Webpack project, so what to do once you find out Amcharts does not support the module pattern.

At first, I reverted back to including a script tag in the HTML file, hoping that my Vue code could grab the reference from the global scope. I’m sure that might have worked, but my new ESLinter gently reminded me that wasn’t a best practice. 😉

It’s worth noting that Amcharts published a quick tutorial on some of this, but it seemed to incur some additional costs to load images, but also didn’t work with ESLint because the AmSerial was never used even though it was imported. ESLint apparently is very picky about things.

So, I found this Github issue that pretty clearly stated Amcharts wasn’t going to support modules any time soon. I tried a few things listed in the comments, but no luck. However, there was one person who mentioned getting it to work with Vue and Webpack by working around with the window object.

AhHa! And sure enough that was the key to getting the best of both worlds. I ended up doing something like the code below:

import 'amcharts3/amcharts/amcharts'

var chart = window.AmCharts.makeChart('chartdiv', {
      config: 'config'
    }

The import statement indicates to Webpack that Amcharts is a dependency and needs to be included in the bundle, but since the Amcharts library doesn’t export anything using modules.export, there is no way to interface with it that way.

After digging into the Amcharts source code, I could easily see in the first few lines that it was modifying the global window object. So, by tapping into the window object myself, I was able to get everything work and pass the ESLint checks.

Maybe there is a better way to go about this type of thing, and I’m sure I’ll learn more as I use Webpack and Vue CLI more, but hopefully this will help some poor soul as they wade through the module murkiness.

 

The post Using AmCharts with Vue and Webpack appeared first on Jeff Everhart.

Outsmarting Google: Generating Download Links with Google App Script

Outsmarting Google: Generating Download Links with Google App Script

For the most part, I love working with Google App Script. The APIs are what you expect them to be. Most of the features are well-documented. Heck, I’ve even tried to build Google Sheets into a small relational database.

But after you’ve been around the block for awhile, you realize there is this odd black market of sorts built into Google App Script and the associated Drive services, things you can do that Google never really meant for you to do, or built in as a feature at some point but forgot about.

This post exposes one of those dirty back alleys you need to generate a download link for Google Documents.

The Scenario

In reality, this should have been a straightforward process. We were trying to loop through a directory structure and print out some data about each file into a Google Sheet. The Google Sheet would then just serve some JSON that a little front end app could consume to allow people to download, copy, or view Google Drive files in a custom way.

All of the looping part worked as expected, but for some reason a previous version of the download link was no longer working.

function execute(){
  //This is the top level folder
  var folderId = "FOLDER_ID_HERE"; 
  var folder = DriveApp.getFolderById(folderId);
  
  var sheet = SpreadsheetApp.getActiveSheet();
  //Append Headers to Sheet
  sheet.appendRow(["File Name", "Parent Folder Name", "URL", "Download Link", "Copy Link"]);
  //Call function to recurse through subfolders
  loopSubFolders(folder, sheet); 
}

function loopSubFolders(parentFolder, sheet){
  var subFolders = parentFolder.getFolders(); 
  listFilesInFolder(subFolders.next(), sheet); 
  while(subFolders.hasNext()){
    listFilesInFolder(subFolders.next(), sheet); 
  }
  
}

function listFilesInFolder(folder, sheet) {
//writes the headers for the spreadsheet
    var contents = folder.getFiles();  
    var cnt = 0;
    var file;

    while (contents.hasNext()) {
        var file = contents.next();
        cnt++;
// writes the various chunks to the spreadsheet- just delete anything you don't want
            data = [
                file.getName(),
                folder.getName(),
                file.getUrl(),
                file.getUrl().split('/edit')[0] + '/export?format=docx', 
                file.getUrl().split('/edit')[0] + '/copy'
            ];

            sheet.appendRow(data);

        

    };
};

Things started to get tricky when logging out the download URL. Google Apps Script makes it easy get the file URL using getUrl, but that is just a link to view the document. After some research, it seemed like getDownloadUrl might do the trick, but alas that didn’t work for any of the Google docs or the random files in the folder.

At some point in this process, I also decided it was a good idea to print out the files as binary blobs in the spreadsheet, which summarily crashed the sheet.

We were able to find some older tutorials that broke down some various link structures, but none of those seemed to work for the current Google Drive setup. At the end of the day, we kind of just started trying things based off the structure of the /copy link until we found something that stuck.

https://docs.google.com/documents/d/YOUR_DOCS_ID_HERE/export?format=pdf

However, this only seems to work for native Google Drive content types, and it needs an export format or otherwise the documents were downloading as HTML. Either way, bound to be useful to someone.

The post Outsmarting Google: Generating Download Links with Google App Script appeared first on Jeff Everhart.

Analyzing and Visualizing Networks

One of the current projects I’m working on involves building out some analytical tools that sit on top of an application that lets students track attendance at extra curricular events for a living and learning program for the daVinci Center. For most of the visualizations, I used amCharts to build out some nice looking and functional charts, but since this data set is pretty unique, I also wanted to explore some of the unique information available that other analytical tools might ignore.

After working together a pretty gnarly SQL query to expose all of the student attendance data through the WordPress REST API, I settled on creating a network graph that shows all of the co-attendance between the students at all of the events.

See the Pen Student Network Analysis by Jeff Everhart (@JEverhart383) on CodePen.

Overall, I was really happy with how this turned out for a few reasons. One of the key tenants of the da Vinci model and its living and learning programs is that the cross-pollination of ideas across disciplines is what leads to innovation.

While the program is open to anyone, at least to my knowledge, they focus on getting students from Business, Engineering, and the Arts to work together. Thus, by encoding those groups in the network graph by using color, we should be able to literally visualize the ways in which students are interacting across those boundaries.

More importantly, it can also help very quickly identify outliers that might not immediately be apparent in other forms, which we can see by the pair of students off in the corner. In a program where collaboration is encouraged, it might be worthwhile to check in on these folks to see what’s up.

Notable Algorithms and Such

Part of the reason that I wanted to do this visualization in the first place was because I’d never written code to piece together the nodes and edges of the network before. As with most things, I decided not to consult the oracles on StackOverflow immediately, and am happy to say that I came up with a working implementation without copying anything from anyone else.

Since this was a undirected graph, meaning there is no directionality associated with the links or edges between nodes, I needed to capture each unique occurrence of a pair of students attending the same event.

Here is what each attendance record looked like in simplified form:

{eventID:511, userEmail:"jeff@awesome.com", ...}

And here is what the finished data structure looked like before feeding it into D3:

let network = {
    "nodes": [{
        "id": "jeff@awesome.com", 
        "group": "Humanities & Sciences"
    }], 
    "links": [{
        "source":"jeff@awesome.com", 
        "target":"everhart@me.com", 
        "value":1
    }]
}

We have a network object, with properties for the nodes and links that both contain array of different objects. Each link object contains a source, target, and value. Since this is undirected, the source and target are sort of arbitrary, and the value specifies the number of times that those two people attended the same event. The value count was integral in helping to weight the links on the force directed graph so that students with more co-attendances are linked more tightly together.

At present, we have only about 50 or so records, but that number will easily quadruple by the end of the semester, so I was interested in making the code used to construct this network diagram as efficient as possible.

In the end, there is one section that amounts to O(n^2) runtime, but I was able to prevent a lot of loops within loops by making using of hash tables, or just plain old objects in JavaScript as my in between data structures.

If you’re interested in looking at that code, you can take a look at the source on GitHub.

 

The post Analyzing and Visualizing Networks appeared first on Jeff Everhart.

Debugging WordPress PHP with VS Code and MAMP

Debugging WordPress PHP with VS Code and MAMP

This is the blog post I wish existed a few weeks ago when I started trying to configure Visual Studio Code to debug PHP and WordPress interactively. Although this setup might not work for everyone depending on your environment, it will save anyone using MAMP already a hell of a lot of time.

The Benefits of Debugging WordPress

One of the reasons I use VS Code is because of its capabilities as an interactive debugger, but how is that different from other types of ‘debugging’?

For a lot of things in development, we can get away with having the program print out some information as it is executing, think console.log for JS or echo, var_dump for PHP, and so on.

But as things get more complicated, you might need to get a better look at what is going on underneath the hood. Enter interactive debugging, which allows you to inspect different variables as the program is executing and step through functions and control flow structures until we decide to release certain breakpoints.

Typically, this helps you resolve issues a lot quicker, especially when something isn’t solved easily by printing out a few key variables.

Enable PHP Debugging in VS Code

VS Code came with support for NodeJS debugging out of the box, so I expected things to be as straightforward for PHP, since it’s more widely used.

Sadly, it was not so.

First, you need to install the PHP Debug extension for VS Code by either running the following command in the command palette or searching for the package in the extensions menu:

ext install php-debug

From here is where things got murky for me. If you install the debugger package, and try to run the debugger it will start throwing errors. Reading the docs for the extension, it says it needs something called Xdebug.

After reading the links to Xdebug on the docs, I embarked on a long trek to download and compile Xdebug from source, configure it with the version of PHP installed on my system (php7), and then it proceeded to keep failing when I tried to have the PHP thread talk back to the debugger.

As it turns out, since I was using MAMP as a local server, Xdebug already comes installed with MAMP, so all you really need to do is instruct MAMP to use Xdebug for the current version of PHP.

I’m sure the other stuff would have been the necessary path if I was using the command line to manage the Apache/MySQL servers.

Enabling Xdebug for MAMP

After spending hours compiling binaries and exporting PATHs, the fix was almost embarrassingly easy. To enable Xdebug on MAMP, we just need to add a few lines to the two php.ini files MAMP references.

Open up the following files in a text editor:

/Applications/MAMP/conf/php[version]/php.ini
 
/Applications/MAMP/bin/php/php[version]/conf/php.ini

Make sure to look in MAMP to see what version of PHP is set as the default, since there are likely dozens of PHP folders inside of the application.

With the files open, add the following lines at the bottom of each file:

[xdebug]
zend_extension="/Applications/MAMP/bin/php/[php_version]/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so"
xdebug.remote_autostart=1
xdebug.remote_enable=1
xdebug.remote_host=localhost
xdebug.remote_port=9000
xdebug.remote_handler=dbgp

At that point, you should be able to restart MAMP, throw a breakpoint in Visual Studio, then hit that breakpoint from your web browser on localhost:8888 or whatever your MAMP config specifies.

Credit to Sean Patterson for the MAMP config details. Hopefully, this will help out some other poor soul by adding to the Google keywords that might lead you to salvation.

The post Debugging WordPress PHP with VS Code and MAMP appeared first on Jeff Everhart.

Debugging WordPress PHP with VS Code and MAMP

Debugging WordPress PHP with VS Code and MAMP

This is the blog post I wish existed a few weeks ago when I started trying to configure Visual Studio Code to debug PHP and WordPress interactively. Although this setup might not work for everyone depending on your environment, it will save anyone using MAMP already a hell of a lot of time.

The Benefits of Debugging WordPress

One of the reasons I use VS Code is because of its capabilities as an interactive debugger, but how is that different from other types of ‘debugging’?

For a lot of things in development, we can get away with having the program print out some information as it is executing, think console.log for JS or echo, var_dump for PHP, and so on.

But as things get more complicated, you might need to get a better look at what is going on underneath the hood. Enter interactive debugging, which allows you to inspect different variables as the program is executing and step through functions and control flow structures until we decide to release certain breakpoints.

Typically, this helps you resolve issues a lot quicker, especially when something isn’t solved easily by printing out a few key variables.

Enable PHP Debugging in VS Code

VS Code came with support for NodeJS debugging out of the box, so I expected things to be as straightforward for PHP, since it’s more widely used.

Sadly, it was not so.

First, you need to install the PHP Debug extension for VS Code by either running the following command in the command palette or searching for the package in the extensions menu:

ext install php-debug

From here is where things got murky for me. If you install the debugger package, and try to run the debugger it will start throwing errors. Reading the docs for the extension, it says it needs something called Xdebug.

After reading the links to Xdebug on the docs, I embarked on a long trek to download and compile Xdebug from source, configure it with the version of PHP installed on my system (php7), and then it proceeded to keep failing when I tried to have the PHP thread talk back to the debugger.

As it turns out, since I was using MAMP as a local server, Xdebug already comes installed with MAMP, so all you really need to do is instruct MAMP to use Xdebug for the current version of PHP.

I’m sure the other stuff would have been the necessary path if I was using the command line to manage the Apache/MySQL servers.

Enabling Xdebug for MAMP

After spending hours compiling binaries and exporting PATHs, the fix was almost embarrassingly easy. To enable Xdebug on MAMP, we just need to add a few lines to the two php.ini files MAMP references.

Open up the following files in a text editor:

/Applications/MAMP/conf/php[version]/php.ini
 
/Applications/MAMP/bin/php/php[version]/conf/php.ini

Make sure to look in MAMP to see what version of PHP is set as the default, since there are likely dozens of PHP folders inside of the application.

With the files open, add the following lines at the bottom of each file:

[xdebug]
zend_extension="/Applications/MAMP/bin/php/[php_version]/lib/php/extensions/no-debug-non-zts-20100525/xdebug.so"
xdebug.remote_autostart=1
xdebug.remote_enable=1
xdebug.remote_host=localhost
xdebug.remote_port=9000
xdebug.remote_handler=dbgp

At that point, you should be able to restart MAMP, throw a breakpoint in Visual Studio, then hit that breakpoint from your web browser on localhost:8888 or whatever your MAMP config specifies.

Credit to Sean Patterson for the MAMP config details. Hopefully, this will help out some other poor soul by adding to the Google keywords that might lead you to salvation.

The post Debugging WordPress PHP with VS Code and MAMP appeared first on Jeff Everhart.

Automate Deployments with GitHub Webhooks

Automate Deployments with GitHub Webhooks

This post is the second post in my new fangled Weekly Workflows section. I took a few requests on Twitter as to what people would like to see here, and Paul Williams, a developer from Texas, expressed some interest in how people use GitHub and workflows around that key technology.

As luck would have it, I’ve been working on building out some tools and workflows using GitHub Webhooks and other parts of our knowledge infrastructure. So, here we go.

Getting Started with GitHub

Although the ALT Lab has been around for a few years, our development group is really in its infancy in a lot of ways. As such, we are trying to figure out how GitHub fits within the larger goals of our organization.

Most of the items in our ALT Lab RVA organizational account are custom WordPress themes or plugins, with a few one-off tools we’ve built for other environments.

As far as context goes, we are not a typically software development shop for a lot of reasons. First, we are typically working with short timelines (think days or weeks, not months), and second, for most projects, there is one person as the lead that might do the bulk of the work.

When we started talking about how GitHub fit into our workflow, it seemed like the main benefit of the service would be the idea of public source control. In other words, although we might not “collaborate” on a project, we all needed access to it in a public way in case something broke in production.

In order to get started without overburdening ourselves with unnecessary structure, we decided to start using only the Master branch of the project in GitHub. We aren’t yet at the level of sophistication where we’d see a ton of benefit from creating new branches, submitting pull requests, or performing code reviews.

However, there is some possibility of a collision when two people create conflicts on the master branch. We have yet to experience that, but to keep everyone in the loop, I integrated our GitHub organization into Slack:

Automate Deployments with GitHub Webhooks

Overall, this has worked so far for us as a cue to perform a fresh pull from master before you start working on a project someone else has touched.

The integration was provided in Slack out of the box, and took me all of two minutes to create and test.

In the future, we might decide that additional process can produce additional benefits, but I’d love to hear from other people, particularly on small, non-standard dev teams to see how they structure this type of workflow.

Continuous Deployment with GitHub Webhooks

Once we started using GitHub to store the latest and greatest versions of whatever we were working on, we ran into issues where the deployed projects in production started to get out of sync with what was in source control.

Before, we had a pretty manual process of using SFTP or SCP to overwrite the old with the new, but that was a manual process and one that is fraught with the potential for human error. After all, there is always the potential for someone to run the wrong command and wipe out the internet.

After looking into a few options, it seemed like the simplest path forward for this was to use GitHub webhooks to trigger an update process on our server. The GitHub docs here are pretty clear, and there is a GUI in GitHub that will let you specify your endpoint and send test pings to practice parsing the payload.

Basically, a webhook is something that just sends out an HTTP POST request to an endpoint you specify when certain conditions are met. At that point, it is up to you to script out the rest of the process.

Here are some code snippets below to get you started:

config.json

This is the file that stores some of the data for the repos you want to be automatically updated. Be sure to check the name of the repo against the “official” name in GitHub. Also note that the commands property should contain all of the commands you want to execute. I imagine this will vary based on your application.

{
  "configurations": [
    {
      "name": "name-of-your-repo", 
      "command": " cd /folder/plugins/your-repo && git pull"
    }

  ]

}

deployment.php

This is the script that does the bulk of the heavy lifting. I tried to add sufficient comments in the code, but if you Google “GitHub Webhooks <your-app-stack>” there will be tons of examples out there. I’m getting more interested in how to use server logs in an intelligent way, so you can see I spend a lot of code logging info from the GitHub request. This was also very helpful when debugging as I wrote the setup.

<?php 

//Pull in the config file which contains the configuration data
//for each repo and the shell commands to excecute when that repo is changed 
$config_file = file_get_contents('config.json');
$config_json = json_decode($config_file, true);  

//First check to see if is a POST, here will will also
//want to incorporate the 
if(isset($_POST)){

  $json = json_decode($_POST['payload'], true); 

  if ($json['action'] == 'published'){
  
//Here I'm setting up a bunch of stuff for some logging. This is helpful as you 
//get started for debuggin, but should also be a good idea so that you can see what kind of 
//traffic is hitting that endpoint 		
    $log = ""; 	
    $github_delivery = $_SERVER['HTTP_X_GITHUB_DELIVERY']; 
    $github_event = $_SERVER['HTTP_X_GITHUB_EVENT']; 
    $user_agent = $_SERVER['HTTP_USER_AGENT']; 
    $repository_name = $json['repository']['name']; 
    $repository_sender = $json['sender']['login'];
    $date = date('l jS \of F Y h:i:s A'); 


    $log .= "\n";
    $log .= "Date: " . $date; 
    $log .= "\n";
    $log .= "Event: ". $github_event;
    $log .= "\n";
    $log .= "Delivery: ". $github_delivery;
    $log .= "\n";
    $log .= "User Agent: ". $user_agent; 
    $log .= "\n"; 
    $log .= "Repoistory Name: ". $repository_name;  
    $log .= "\n"; 
    $log .= "Sender Name: ". $repository_sender;  
    $log .= "\n"; 

    file_put_contents('deploy-log.txt', $log, FILE_APPEND);

//Once we've parsed the payload, loop through the available environments and use shell_exec to 
//execute the shell commands you've associated with the environment 		
    foreach ($config_json['configurations'] as $config) {
      if ($config['name'] == $repository_name){

        shell_exec($config['command']); 


      }
    }


  }

}



?>

That’s about all she wrote for this one. After struggling with some PHP nuance I’m not familiar with, this was a fairly straightforward thing to do. There is still some manual process here because we have to add another record to config.json, but I can deal with one manual step for an automated future.

Things to Ponder

While this is going to do wonders for our workflow, there are lots of spaces for you to make this process your own and lots of other things that might need to be tweaked to fit in other architectures.

First, we had to make decisions about what GitHub events would trigger the webhook. In the end, we decided to rebuild our stuff anytime a new release is tagged in GitHub. Part of that reasoning is that individual commits seemed too aggressive, while the semantic versioning enforced by the release structure would actually help us better track down bugs as things in the app change.

However, it’s not an all or nothing. GitHub will let you specify the events that trigger a webhook, or you can listen for everything and have your app react based on events that you defined. In our setup, we might have one script listen for changes on dozens of repos all in different stages of development. It might be cool to have a few listen for the ‘release’ or ‘publish’ event, but also have some work in progress stuff listen for just commits.

The world is your oyster.

Another thing to think about is how this pattern would fit in if we were using something like Node/Express or Flask/Python, or any other number of frameworks or runtimes. People give PHP a lot of crap, but even Slack’s engineering team uses it for their backend code, citing developer productivity as a benefit.

As someone who’s worked extensively in the Node ecosystem, where server restarts are required, and in C# .NET, where all code is complied at runtime, I could see the process including some additional steps. Replacing a PHP file then making a new request is a pretty sweet pattern in comparison.

I’m really interested in hearing what workflows like this look like for other people, especially if you are on a small team or doing non-standard (meaning not consumer software) work.

The post Automate Deployments with GitHub Webhooks appeared first on Jeff Everhart.

Automate Deployments with GitHub Webhooks

Automate Deployments with GitHub Webhooks

This post is the second post in my new fangled Weekly Workflows section. I took a few requests on Twitter as to what people would like to see here, and Paul Williams, a developer from Texas, expressed some interest in how people use GitHub and workflows around that key technology.

As luck would have it, I’ve been working on building out some tools and workflows using GitHub Webhooks and other parts of our knowledge infrastructure. So, here we go.

Getting Started with GitHub

Although the ALT Lab has been around for a few years, our development group is really in its infancy in a lot of ways. As such, we are trying to figure out how GitHub fits within the larger goals of our organization.

Most of the items in our ALT Lab RVA organizational account are custom WordPress themes or plugins, with a few one-off tools we’ve built for other environments.

As far as context goes, we are not a typically software development shop for a lot of reasons. First, we are typically working with short timelines (think days or weeks, not months), and second, for most projects, there is one person as the lead that might do the bulk of the work.

When we started talking about how GitHub fit into our workflow, it seemed like the main benefit of the service would be the idea of public source control. In other words, although we might not “collaborate” on a project, we all needed access to it in a public way in case something broke in production.

In order to get started without overburdening ourselves with unnecessary structure, we decided to start using only the Master branch of the project in GitHub. We aren’t yet at the level of sophistication where we’d see a ton of benefit from creating new branches, submitting pull requests, or performing code reviews.

However, there is some possibility of a collision when two people create conflicts on the master branch. We have yet to experience that, but to keep everyone in the loop, I integrated our GitHub organization into Slack:

Automate Deployments with GitHub Webhooks

Overall, this has worked so far for us as a cue to perform a fresh pull from master before you start working on a project someone else has touched.

The integration was provided in Slack out of the box, and took me all of two minutes to create and test.

In the future, we might decide that additional process can produce additional benefits, but I’d love to hear from other people, particularly on small, non-standard dev teams to see how they structure this type of workflow.

Continuous Deployment with GitHub Webhooks

Once we started using GitHub to store the latest and greatest versions of whatever we were working on, we ran into issues where the deployed projects in production started to get out of sync with what was in source control.

Before, we had a pretty manual process of using SFTP or SCP to overwrite the old with the new, but that was a manual process and one that is fraught with the potential for human error. After all, there is always the potential for someone to run the wrong command and wipe out the internet.

After looking into a few options, it seemed like the simplest path forward for this was to use GitHub webhooks to trigger an update process on our server. The GitHub docs here are pretty clear, and there is a GUI in GitHub that will let you specify your endpoint and send test pings to practice parsing the payload.

Basically, a webhook is something that just sends out an HTTP POST request to an endpoint you specify when certain conditions are met. At that point, it is up to you to script out the rest of the process.

Here are some code snippets below to get you started:

config.json

This is the file that stores some of the data for the repos you want to be automatically updated. Be sure to check the name of the repo against the “official” name in GitHub. Also note that the commands property should contain all of the commands you want to execute. I imagine this will vary based on your application.

{
  "configurations": [
    {
      "name": "name-of-your-repo", 
      "command": " cd /folder/plugins/your-repo && git pull"
    }

  ]

}

deployment.php

This is the script that does the bulk of the heavy lifting. I tried to add sufficient comments in the code, but if you Google “GitHub Webhooks <your-app-stack>” there will be tons of examples out there. I’m getting more interested in how to use server logs in an intelligent way, so you can see I spend a lot of code logging info from the GitHub request. This was also very helpful when debugging as I wrote the setup.

<?php 

//Pull in the config file which contains the configuration data
//for each repo and the shell commands to excecute when that repo is changed 
$config_file = file_get_contents('config.json');
$config_json = json_decode($config_file, true);  

//First check to see if is a POST, here will will also
//want to incorporate the 
if(isset($_POST)){

  $json = json_decode($_POST['payload'], true); 

  if ($json['action'] == 'published'){
  
//Here I'm setting up a bunch of stuff for some logging. This is helpful as you 
//get started for debuggin, but should also be a good idea so that you can see what kind of 
//traffic is hitting that endpoint 		
    $log = ""; 	
    $github_delivery = $_SERVER['HTTP_X_GITHUB_DELIVERY']; 
    $github_event = $_SERVER['HTTP_X_GITHUB_EVENT']; 
    $user_agent = $_SERVER['HTTP_USER_AGENT']; 
    $repository_name = $json['repository']['name']; 
    $repository_sender = $json['sender']['login'];
    $date = date('l jS \of F Y h:i:s A'); 


    $log .= "\n";
    $log .= "Date: " . $date; 
    $log .= "\n";
    $log .= "Event: ". $github_event;
    $log .= "\n";
    $log .= "Delivery: ". $github_delivery;
    $log .= "\n";
    $log .= "User Agent: ". $user_agent; 
    $log .= "\n"; 
    $log .= "Repoistory Name: ". $repository_name;  
    $log .= "\n"; 
    $log .= "Sender Name: ". $repository_sender;  
    $log .= "\n"; 

    file_put_contents('deploy-log.txt', $log, FILE_APPEND);

//Once we've parsed the payload, loop through the available environments and use shell_exec to 
//execute the shell commands you've associated with the environment 		
    foreach ($config_json['configurations'] as $config) {
      if ($config['name'] == $repository_name){

        shell_exec($config['command']); 


      }
    }


  }

}



?>

That’s about all she wrote for this one. After struggling with some PHP nuance I’m not familiar with, this was a fairly straightforward thing to do. There is still some manual process here because we have to add another record to config.json, but I can deal with one manual step for an automated future.

Things to Ponder

While this is going to do wonders for our workflow, there are lots of spaces for you to make this process your own and lots of other things that might need to be tweaked to fit in other architectures.

First, we had to make decisions about what GitHub events would trigger the webhook. In the end, we decided to rebuild our stuff anytime a new release is tagged in GitHub. Part of that reasoning is that individual commits seemed too aggressive, while the semantic versioning enforced by the release structure would actually help us better track down bugs as things in the app change.

However, it’s not an all or nothing. GitHub will let you specify the events that trigger a webhook, or you can listen for everything and have your app react based on events that you defined. In our setup, we might have one script listen for changes on dozens of repos all in different stages of development. It might be cool to have a few listen for the ‘release’ or ‘publish’ event, but also have some work in progress stuff listen for just commits.

The world is your oyster.

Another thing to think about is how this pattern would fit in if we were using something like Node/Express or Flask/Python, or any other number of frameworks or runtimes. People give PHP a lot of crap, but even Slack’s engineering team uses it for their backend code, citing developer productivity as a benefit.

As someone who’s worked extensively in the Node ecosystem, where server restarts are required, and in C# .NET, where all code is complied at runtime, I could see the process including some additional steps. Replacing a PHP file then making a new request is a pretty sweet pattern in comparison.

I’m really interested in hearing what workflows like this look like for other people, especially if you are on a small team or doing non-standard (meaning not consumer software) work.

The post Automate Deployments with GitHub Webhooks appeared first on Jeff Everhart.

Contact us

Academic Learning Transformation Lab - ALT Lab
1000 Floyd Ave, Suite 4102 | Richmond, Virginia 23284
altlab@vcu.edu | 804-827-5181

Last updated: September 26, 2017

Virginia Commonwealth University