Author Archives: Jeff

Aggregating Data Hypotheses with Firebase

Aggregating Data Hypotheses with Firebase

This project builds off of some work that Tom and Matt did over last summer for an eco techniques course focused on open science, data visualization, and experiment design. We were trying to replicate an interesting interactive published by the New York Times on the Obama presidency. The main crux of this interactive is the idea of expectation vs. reality, how you think something is vs. what it actually is.

 

While this sentiment is important in politics, it is also important in science as well as you make, test, and prove/disprove hypotheses. The basic interaction is quite simple; you see a portion of a chart, then you fill in the rest of the chart based on your assumptions (your hypothesis).

 

Once you’ve taken your position, the rest of the chart fills in with the correct data so you can see how close you are to reality.

 

However, we’ve talked about taking this one step further and projecting another layer of aggregation onto the chart. Right now, you can see how close you are to reality, but how close were you to the other people that went through the same experience. Is everyone underestimating or overestimating a position based on some variables, or is the trend clear to everyone interacting with this graphic?

 

How Does This Work?

 

We start off with a simple chart drawn in D3 with a large portion of the data path clipped to hide it from the user. The black series here is data that we have from our Flux Tower source, so in other words this is ‘actual’ data.

 

 

From here, the user can interact with the chart and start to draw their hypothesis. Essentially, they are trying to guess where the data series will go based on the short snippet of the actual data they can see. As the user draws their line, a green dashed line gets drawn to the chart.

 

Aggregating Data Hypotheses with Firebase

 

As the user draws their hypothesis, our JavaScript code is looking for the user guess to hold a specific amount of data points. That way, we don’t jump the gun and drawn the actual data or the aggregate hypothesis before the user has created a full guess. 

 

Once we have those data points in the JS function, D3 animates the drawing of both the rest of the ‘actual’ data and the aggregate hypothesis data that is stored in Firebase. 

 

Aggregating Data Hypotheses with Firebase

 

Overall, there is some pretty cool code in this project, particularly the clipped path D3 chart stuff that Tom and Matt wrote last fall. Really, this addition piggybacks on their code and integrates Firebase to easily store the data for the guesses. None of that code is really crazy, but it is all nonetheless available in GitHub if you’re interested in looking at the guts of this project.

The post Aggregating Data Hypotheses with Firebase appeared first on Jeff Everhart.

Aggregating Data Hypotheses with Firebase

Aggregating Data Hypotheses with Firebase

This project builds off of some work that Tom and Matt did over last summer for an eco techniques course focused on open science, data visualization, and experiment design. We were trying to replicate an interesting interactive published by the New York Times on the Obama presidency. The main crux of this interactive is the idea of expectation vs. reality, how you think something is vs. what it actually is.

 

While this sentiment is important in politics, it is also important in science as well as you make, test, and prove/disprove hypotheses. The basic interaction is quite simple; you see a portion of a chart, then you fill in the rest of the chart based on your assumptions (your hypothesis).

 

Once you’ve taken your position, the rest of the chart fills in with the correct data so you can see how close you are to reality.

 

However, we’ve talked about taking this one step further and projecting another layer of aggregation onto the chart. Right now, you can see how close you are to reality, but how close were you to the other people that went through the same experience. Is everyone underestimating or overestimating a position based on some variables, or is the trend clear to everyone interacting with this graphic?

 

How Does This Work?

 

We start off with a simple chart drawn in D3 with a large portion of the data path clipped to hide it from the user. The black series here is data that we have from our Flux Tower source, so in other words this is ‘actual’ data.

 

 

From here, the user can interact with the chart and start to draw their hypothesis. Essentially, they are trying to guess where the data series will go based on the short snippet of the actual data they can see. As the user draws their line, a green dashed line gets drawn to the chart.

 

Aggregating Data Hypotheses with Firebase

 

As the user draws their hypothesis, our JavaScript code is looking for the user guess to hold a specific amount of data points. That way, we don’t jump the gun and drawn the actual data or the aggregate hypothesis before the user has created a full guess. 

 

Once we have those data points in the JS function, D3 animates the drawing of both the rest of the ‘actual’ data and the aggregate hypothesis data that is stored in Firebase. 

 

Aggregating Data Hypotheses with Firebase

 

Overall, there is some pretty cool code in this project, particularly the clipped path D3 chart stuff that Tom and Matt wrote last fall. Really, this addition piggybacks on their code and integrates Firebase to easily store the data for the guesses. None of that code is really crazy, but it is all nonetheless available in GitHub if you’re interested in looking at the guts of this project.

The post Aggregating Data Hypotheses with Firebase appeared first on Jeff Everhart.

Storytelling with Scrolling Map Background

For a recent project, we were helping a group from the Environmental Sciences department translate some of their academic research on a particular type of bird into a more general narrative targeted towards a general audience.

As a part of this process, we wanted to create a section of our companion site that illustrates the annual migration pattern of the Prothonotary Warbler. The migration is important because of the wide range the birds travel, starting in the North American south and ending in the Panamanian mangroves during winter.

In preparation for our part of the work, we looked at other examples of compelling digital storytelling and found a lot of great examples at The Pudding. Most of their most polished pieces use an interesting sticky scrolling pattern, where the screen is split into equal parts and as the user scrolls through textual explanations a corresponding data visualization of some kind is updated based on the user’s position.

Matt worked up some great mockups that we were using to build out our interactive components, and his mockup used a full screen background that outlines this migration pattern. After looking at this, it seemed like we should be able to make this interactive in the same way The Pudding does using some simple map libraries.

I based my code in part on the examples describing the usage of the Scrollama JS library and found everything pretty intuitive. There isn’t anything super fancy here with the code, so you can check that out in the embedded code pen below, but I figured this was worth writing about because it seems like an interesting take on an already popular pattern in digital storytelling.

See the Pen Scroll Map Background by Jeff Everhart (@JEverhart383) on CodePen.

The post Storytelling with Scrolling Map Background appeared first on Jeff Everhart.

Simple is Always Better

One of the hardest things to do as a developer is to resist the urge to add additional complexity where it does not add value to the overall solution. As technologists, we tend to think this stuff is cool, and so sometimes we fall victim to this trap. Let me illustrate with a recent example.

We have two environments for our large WordPress multisite installation, a production environment and a development environment, with dev being an exact clone of production. However, the problem we’ve been struggling with for some time is that WordPress uses that database to generate links for the individual sites using the siteurl and home options in the wp_options table.

Since dev is a copy of prod, it leads to links that point back to our production environment, which is not great when you are testing things. In most cases, a simple search and replace plugin would help solve this issue, but none of them are set-up to work across a sharded database, and our multisite installation is spread across 256 separate MySQL databases.

Possible Solutions

As Tom and I talked about different ways to handle this, most of the solutions we could think of involved pretty complex and low-level solutions. We talked about writing a pure PHP script, outside of the WordPress environment, to loop through the databases and their tables, updating all of the options fields. We talked about writing something similar just using SQL, but eventually I settled on writing a small plugin to use the WordPress functionality to loop through and update all of the site options.

Initially, I tried one loop with all 30K sites, but it promptly crashed the dev environment. From there I implemented some pagination to update 100 sites at a time. You can see the entirety of the plugin code below:

add_action('network_admin_menu', 'add_rampages_dev_url_replace_menu');
function add_rampages_dev_url_replace_menu() {
    add_menu_page(
        "Rampages Dev URL Replace",
        "Rampages Dev URL Replace",
        'manage_network',
        'rampages-dev-url-replace',
        'rampages_dev_url_replace_render_menu'
    );
}


function rampages_dev_url_replace_render_menu(){
    $offset = isset($_GET['offset']) ? $_GET['offset'] : 0;
    echo 'Rampages Dev Replace URL';
    $sites = rampages_dev_url_get_sites($offset * 100);

    //Check to make sure we are not at the end of the sites list
    if (sizeof($sites)> 0){

        foreach($sites as $site){
            $blog_id = $site->blog_id;
            switch_to_blog($blog_id);
            $siteurl = '';
            $home = '';
            if(get_option('siteurl')){
                $siteurl = get_option('siteurl');
                update_option('siteurl', preg_replace('/rampages.us/', 'rampagesdev.reclaimhosting.com', $siteurl));
                $newsiteurl = get_option('siteurl');
            }
            if (get_option('home')){
                $home = get_option('home');
                update_option('home', preg_replace('/rampages.us/', 'rampagesdev.reclaimhosting.com', $home));
                $newhome = get_option('home');
            }
            echo "</br>". $blog_id . " " . $siteurl . " " . $home;
            restore_current_blog();
        }
        $offset++;
        var_dump($offset);
        echo "<script type='text/javascript'>window.location.href='https://rampagesdev.reclaimhosting.com/wp-admin/network/admin.php?page=rampages-dev-url-replace". "&offset=". $offset ."';</script>";
    } else {
        echo "Youre all done!";
    }
    //redirect to ?offset=2
}


function rampages_dev_url_get_sites($offset){
    $args = array(
        'offset' => $offset,
    );
    $sites = get_sites($args);
    return $sites;

}

Overall, still pretty simple as far as plugins go, but right about the time I got this working, Tom came up with an even simpler solution using filters:

function replace_siteurl($val) {
  $clean = preg_replace('/rampages.us/', 'rampagesdev.reclaimhosting.com', $val);
    return $clean;
}
add_filter('option_siteurl', 'replace_siteurl');
add_filter('option_home', 'replace_siteurl');

Both technically work and do the same thing, but it’s worth talking about their approach to addressing the underlying problem.

The first and longer plugin attempts to address the issue from a really technical perspective. The data in the database in incorrect and we need to change it, so we make a solution that does just that. In fact, all of the other possible early solutions we came up with involved manipulating the underlying data.

However, since this is a dev environment that we mainly use to test updates in production, we didn’t need for the data to be perfect. Really we were just looking for a way to navigate around the dev environment without accidentally ending up in production, which the filters address in ~6 lines of code.

The Lesson

In the development world, simpler is generally better. Full stop. It also helps if you take some time to break down the problem into it’s simplest possible variation. Had we phrased things differently in our initial conversations around solutions, maybe we would have realized that the real problem wasn’t that the data was wrong, but that we needed a way to change the links being echoed in the dev environment.

It’s hard to say whether these linguistic nuances would have lead us any sooner to WordPress filters, but I’d like to think that a thorough reframing of the problem helps us access different paths forward.

The post Simple is Always Better appeared first on Jeff Everhart.

Simple is Always Better

One of the hardest things to do as a developer is to resist the urge to add additional complexity where it does not add value to the overall solution. As technologists, we tend to think this stuff is cool, and so sometimes we fall victim to this trap. Let me illustrate with a recent example.

We have two environments for our large WordPress multisite installation, a production environment and a development environment, with dev being an exact clone of production. However, the problem we’ve been struggling with for some time is that WordPress uses that database to generate links for the individual sites using the siteurl and home options in the wp_options table.

Since dev is a copy of prod, it leads to links that point back to our production environment, which is not great when you are testing things. In most cases, a simple search and replace plugin would help solve this issue, but none of them are set-up to work across a sharded database, and our multisite installation is spread across 256 separate MySQL databases.

Possible Solutions

As Tom and I talked about different ways to handle this, most of the solutions we could think of involved pretty complex and low-level solutions. We talked about writing a pure PHP script, outside of the WordPress environment, to loop through the databases and their tables, updating all of the options fields. We talked about writing something similar just using SQL, but eventually I settled on writing a small plugin to use the WordPress functionality to loop through and update all of the site options.

Initially, I tried one loop with all 30K sites, but it promptly crashed the dev environment. From there I implemented some pagination to update 100 sites at a time. You can see the entirety of the plugin code below:

add_action('network_admin_menu', 'add_rampages_dev_url_replace_menu');
function add_rampages_dev_url_replace_menu() {
    add_menu_page(
        "Rampages Dev URL Replace",
        "Rampages Dev URL Replace",
        'manage_network',
        'rampages-dev-url-replace',
        'rampages_dev_url_replace_render_menu'
    );
}


function rampages_dev_url_replace_render_menu(){
    $offset = isset($_GET['offset']) ? $_GET['offset'] : 0;
    echo 'Rampages Dev Replace URL';
    $sites = rampages_dev_url_get_sites($offset * 100);

    //Check to make sure we are not at the end of the sites list
    if (sizeof($sites)> 0){

        foreach($sites as $site){
            $blog_id = $site->blog_id;
            switch_to_blog($blog_id);
            $siteurl = '';
            $home = '';
            if(get_option('siteurl')){
                $siteurl = get_option('siteurl');
                update_option('siteurl', preg_replace('/rampages.us/', 'rampagesdev.reclaimhosting.com', $siteurl));
                $newsiteurl = get_option('siteurl');
            }
            if (get_option('home')){
                $home = get_option('home');
                update_option('home', preg_replace('/rampages.us/', 'rampagesdev.reclaimhosting.com', $home));
                $newhome = get_option('home');
            }
            echo "</br>". $blog_id . " " . $siteurl . " " . $home;
            restore_current_blog();
        }
        $offset++;
        var_dump($offset);
        echo "<script type='text/javascript'>window.location.href='https://rampagesdev.reclaimhosting.com/wp-admin/network/admin.php?page=rampages-dev-url-replace". "&offset=". $offset ."';</script>";
    } else {
        echo "Youre all done!";
    }
    //redirect to ?offset=2
}


function rampages_dev_url_get_sites($offset){
    $args = array(
        'offset' => $offset,
    );
    $sites = get_sites($args);
    return $sites;

}

Overall, still pretty simple as far as plugins go, but right about the time I got this working, Tom came up with an even simpler solution using filters:

function replace_siteurl($val) {
  $clean = preg_replace('/rampages.us/', 'rampagesdev.reclaimhosting.com', $val);
    return $clean;
}
add_filter('option_siteurl', 'replace_siteurl');
add_filter('option_home', 'replace_siteurl');

Both technically work and do the same thing, but it’s worth talking about their approach to addressing the underlying problem.

The first and longer plugin attempts to address the issue from a really technical perspective. The data in the database in incorrect and we need to change it, so we make a solution that does just that. In fact, all of the other possible early solutions we came up with involved manipulating the underlying data.

However, since this is a dev environment that we mainly use to test updates in production, we didn’t need for the data to be perfect. Really we were just looking for a way to navigate around the dev environment without accidentally ending up in production, which the filters address in ~6 lines of code.

The Lesson

In the development world, simpler is generally better. Full stop. It also helps if you take some time to break down the problem into it’s simplest possible variation. Had we phrased things differently in our initial conversations around solutions, maybe we would have realized that the real problem wasn’t that the data was wrong, but that we needed a way to change the links being echoed in the dev environment.

It’s hard to say whether these linguistic nuances would have lead us any sooner to WordPress filters, but I’d like to think that a thorough reframing of the problem helps us access different paths forward.

The post Simple is Always Better appeared first on Jeff Everhart.

Reflections on Data Visualization in Observable

Reflections on Data Visualization in Observable

Continuing my exploration of data visualization tools, I circled back to where it all started for me: D3. For those of you involved in developing data-driven apps for the web, you most likely have a love/hate relationship with D3. You look at all of the cool examples and think, “Wow! That looks great,” but then try to pull your hair out once you actually try to make one yourself. At least that has been my pattern.

And after almost five years of this feeling, I can safely say not a lot has changed.

Nothing seems any easier from D3 v3 to v5, but maybe my tolerance for pain has gone up. Either way, only part of my desire here is to actually code bespoke data visualizations. In most cases, I’m just looking for a more compelling way to tell data-driven stories.

In a recent post, I explored some tools for data visualization in AWS, but Quicksight, while easy to use, wasn’t an ideal platform for sharing data analysis with the general public. Well, enter Observable, a Jupyter Notebook-like interactive data journal that uses JavaScript instead of Python. You can check out my evolving experiment here, but also check out some of the examples as they illustrate the power of the platform better than I ever could.

Below you’ll find a quick list of pros/cons based on my experience using other tools.

Pros of Observable

Overall, I was really impressed by this tool and what it made possible. In terms of creating readable data products, this provides a lot of support for what we call ‘explorable explanations.’ If you are looking for a full blown dashboard, Observable might not be the best platform, but if you want some expository text accompanied by JS-based data visualizations or maps, this is a great way to publish some stuff quickly.

You Can Make Complex Things

Observable lets you get as complicated as you want, as you can see by the graphic below. This was a set of geoJSON coordinates projected into SVG with a proportional symbol layer of census tract population. What a mouthful, but still something Observable is able to process and quickly render to the page. 

You Don’t Have to Use D3

Although most of the examples you can find use D3 to create visualizations, Observable is pretty agnostic when it comes to what types of code you want to run in your notebooks. As long as the packages are built using some form of JS modules or are available via NPM, you can use them in your work on the site. Check out this example using Google Maps for some details on how to make this happen.

You Can Create Beautiful Documents

In the past, most of my visualizations end up being one-off visualizations that exist on a webpage that may be fractured across on of my subdomain or various properties. See this visualization of missing children reports for an example. Usually, they are just the visualization itself without any context or expository text. While I could certainly add some additional text to each of these HTML pages, it isn’t always high up on my list. Now, with Observable this becomes a built-in format, so it makes it much more likely, for me at least, that I’ll include this extra context to round out my data explorable graphics.

The Process Is Clear

For most projects, the user can really only see the end result and not the process, unless you write some sort of accompanying blog post. For data science things, especially when we’re dealing with some type of data that needs to be cleaned, the process of cleaning and transforming the data can be as important as result itself. Using the Observable notebook, if we make those transformations as a part of the code in the notebook, people can see the process by which you arrived at your final product.

Cons of Observable

While there are obviously a lot of benefits of the Observable notebook, I find it only fitting to talk about some of the things I struggled with in creating my examples.

Difficulty Using Developer Tools

I’m sure this con would be diminished for someone more fluent with D3, or if I used a library or framework I can write with less trial and error, but I found prototyping somewhat more difficult than just using a local server. Part of this has to do with the fact that you are coding in an application that is also running in the browser, so things you log using console.log get mixed in with all of the things that Observable is already logging.

I have the same criticism with CodePen, but I also understand there is really no way around this given the way the technology works. If you are looking to simplify the development process, in terms of inspecting elements or using the JavaScript console, these types of tools make that piece difficult.

You Might Need to Up Your JavaScript Game

Over that last year, I’ve been trying to incorporate more ES6 and functional programming features into the JS code I write, namely using array methods like map, filter, reduce and handy things like template literals. As I started messing with more complex Vue apps, I had to learn more about modules and such. There are a lot of places where Observable relies on these cutting edge techniques, and that’s not to say there isn’t a way to develop “old school,” but rather that non of the examples are written that way.

For example, I had a tough time including a library that wasn’t written using the module pattern. In a regular visualization, I would have just included the script tag, and that would have been that. Not so in Observable. Now, again, that doesn’t mean you can’t, just that it wasn’t immediately apparent how to do so.

Conclusion

Overall, this was a really fun experiment that got me reacquainted with the newest version of D3 and added another tool to my tool belt. I think using Observable with a high-level library like Vega lite would be super productive. I’ll continue to play around, experiment, and report back to you, the good people of the internet.

The post Reflections on Data Visualization in Observable appeared first on Jeff Everhart.

Digging into the Gutenberg Editor

Digging into the Gutenberg Editor

After a decent amount of foot shuffling, I finally decided to step up and dig into the Gutenberg editor that will ship with WordPress 5.0, whenever that is ready. There has been a lot of chatter in the WP ecosystem over the last year or so about when this update will be ready, how it will upend or improve existing workflows, and how we as developers can start to work with the new editing experience.

From a personal standpoint, there are a few reasons why I have waited this long to start playing around with the feature that will be the future of WordPress. First, I’ve been feeling a bit of JS framework fatigue lately. Gutenberg is built on top of React, which I have the least experience with, so even the thought of toying around in a serious way necessitated some additional self-learning. Second, all of the examples I’ve seen of blocks are built using Webpack and a slew of other dependencies.

What I really wanted was a bare bones tutorial, sans all of the aforementioned complexities. So, partially because I couldn’t find what I wanted, and partially because making things is how I learn, I’ve created a GitHub repo with some lowest common denominator examples. A few of these are modified versions of what can be found in some of the official docs, but I thought it would be nice to collect some examples for local development.

The Basics of Blocks

It turns out that Gutenberg blocks have a lot going for them. At the basic level, they are pretty simple to create as a part of themes or plugins, as I’ll show later in this post. At the same time, the way they store data as semantic HTML in the post itself is in some ways very preferable to making endless calls to get custom meta fields as a pattern for creating structured content.

At the core of block creation, we need to do two things in our plugin/theme files. We need to create a JavaScript file for the block to use when it is edited and saved, and then enqueue that JS file to make the block available.

Let’s take a look at the minimum JS required to register a simple block:

// wp.element and wp.blocks are both parts of the
// WP JS API, and we need both to create parts of your block
// and register it for use in the editor
var el = wp.element.createElement,
  registerBlockType = wp.blocks.registerBlockType,
  blockStyle = { backgroundColor: '#fff', border: '2px solid #900', color: '#000', padding: '20px' };


// We call the registerBlockType function to register your block type
// for use in the editor. We need to namespace our blocks 'your-namespace/your-block'
// to avoid collisions between our blocks and others	
registerBlockType( 'cornerstone-block/test-block', {
  title: 'Cornerstone Block 1',
  // icon can be a dash icon or a path to an SVG/PNG
  icon: 'universal-access-alt',
  category: 'layout',
  // Here we define the edit and save methods. Both of these should return a call to 
  // wp.element.createElement with the content we want rendered in our block on edit and save
  edit: function() {
    return el( 'p', { style: blockStyle }, 'This is what appears in the editor.' );
  },

  save: function() {
    return el( 'p', { style: blockStyle }, 'This is what gets saved into post content.' );
  },
} );

Hopefully, most of the comments are enough, but the gist is that we call wp.blocks.registerBlockType and pass it a configuration object that contains save and edit methods, both of which return an element created using wp.element.createElement, which is really a wrapper around React functionality.

One of the things I like already about the block pattern is that I’m not forced to use React. I can use the abstractions provided by the WP JS API instead of dealing with React, at least for most easy things like this. I’m sure I’ll dig into React more over the coming weeks, but I like that I can workaround it for as long as possible.

Enqueue and Register Block in PHP

Now that we have a good sense for the basic JS structure you need to create a block, we need to enqueue the script for the editor and register the block within the core of WP’s PHP. If you’ve done any plugin or theme development in the past, this should look pretty familiar to you already:

function cornerstone_boilerplate_block() {

  // Here we register the script as we would normally, but make
  // sure to pass wp-blocks and wp-element into the dependencies 
  // argument at the end. This tells WP that our script relies on the others  
  wp_register_script(
    'cornerstone-block-1-js',
    plugins_url( 'cornerstone-block-1.js', __FILE__ ),
    array( 'wp-blocks', 'wp-element' )
  );

  // Here we actually register the block with WP, again using our namespacing
  // We also specify the editor script to be used in the Gutenberg interface 
  register_block_type( 'cornerstone-block/cornerstone-block-1', array(
    'editor_script' => 'cornerstone-block-1-js',
  ) );

}
add_action( 'init', 'cornerstone_boilerplate_block' );

Overall, this is still a pretty familiar pattern to most WP devs, and there are even some ways we could make PHP do some rendering here of block content as well, which makes this feel like an easier-to-use shortcode interface. I’ll talk about that in future posts.

What You See Is What You Really Get

One of the largest pushes for a switch to the Gutenberg interface has to do with what can sometimes be a black box of the WP editor interface. Even though TinyMCE is billed as a WYSIWYG editor, in most cases users needed preview the page to see what it would look like styled and rendered.

Gutenberg helps solve some of that problem, by rendering rich HTML directly into the editing interface. For example, this is how our tiny test block is rendered in the editor.

If we look back at our original JS code, we can see that this is the element and content we returned with the edit method attached to our block.

edit: function() {
    return el( 'p', { style: blockStyle }, 'This is what appears in the editor.' );
  },

In a similar fashion, when we update or save the post, the content in the editor switches to whatever element we’ve returned from our save method. This allows us to specify an editing experience for our blocks that is different from how the content is ultimately displayed on the front end.

Our test block uses the following code in our save method:

save: function() {
    return el( 'p', { style: blockStyle }, 'This is what gets saved into post content.' );
  }

And its output looks like this:

Digging into the Gutenberg Editor

Overall, this provides a pretty clear pattern to use when you create blocks. Obviously, we can and will begin to create more sophisticated editing interfaces that require configuration and fuller saved content based on those variables, but this is a good start.

How Blocks Are Saved

Now, you might be asking yourself where some of this configuration data for a custom block gets saved. Most of us have used custom fields to store additional data that we need to construct a custom part of a page, post, or template, but Gutenberg turns that method on its head in a lot of ways. I’m not saying custom fields are going away, or will lose value, but Gutenberg can actually save and retrieve some configuration values from the blocks themselves.

First to understand how this works, lets look at how the content of blocks gets saved into the post content in the database.

Digging into the Gutenberg Editor

Gutenberg uses a system of comments to bracket all of the HTML produced when you save a block. If we look closer at the actual output, there are also classes added to the block content that we can hook into for CSS style.

By structuring the markup in this way, with all of the additional namespacing provided by the comments and the classes, Gutenberg can grab attributes of content from these DOM elements directly, which means that certain types of data can be stored in the blocks themselves as a part of regular post data.

Overall, its a very cool concept, and we’ll look at that in further details in future block examples, but it should make life a bit easier for developers as well. Meaning we can create a block, define its output, then just deal with it like any other post content. No need to do lots of calls to get_post_meta to construct all of the data we need to render a template.

 

 

 

 

The post Digging into the Gutenberg Editor appeared first on Jeff Everhart.

Digging into the Gutenberg Editor

Digging into the Gutenberg Editor

After a decent amount of foot shuffling, I finally decided to step up and dig into the Gutenberg editor that will ship with WordPress 5.0, whenever that is ready. There has been a lot of chatter in the WP ecosystem over the last year or so about when this update will be ready, how it will upend or improve existing workflows, and how we as developers can start to work with the new editing experience.

From a personal standpoint, there are a few reasons why I have waited this long to start playing around with the feature that will be the future of WordPress. First, I’ve been feeling a bit of JS framework fatigue lately. Gutenberg is built on top of React, which I have the least experience with, so even the thought of toying around in a serious way necessitated some additional self-learning. Second, all of the examples I’ve seen of blocks are built using Webpack and a slew of other dependencies.

What I really wanted was a bare bones tutorial, sans all of the aforementioned complexities. So, partially because I couldn’t find what I wanted, and partially because making things is how I learn, I’ve created a GitHub repo with some lowest common denominator examples. A few of these are modified versions of what can be found in some of the official docs, but I thought it would be nice to collect some examples for local development.

The Basics of Blocks

It turns out that Gutenberg blocks have a lot going for them. At the basic level, they are pretty simple to create as a part of themes or plugins, as I’ll show later in this post. At the same time, the way they store data as semantic HTML in the post itself is in some ways very preferable to making endless calls to get custom meta fields as a pattern for creating structured content.

At the core of block creation, we need to do two things in our plugin/theme files. We need to create a JavaScript file for the block to use when it is edited and saved, and then enqueue that JS file to make the block available.

Let’s take a look at the minimum JS required to register a simple block:

// wp.element and wp.blocks are both parts of the
// WP JS API, and we need both to create parts of your block
// and register it for use in the editor
var el = wp.element.createElement,
  registerBlockType = wp.blocks.registerBlockType,
  blockStyle = { backgroundColor: '#fff', border: '2px solid #900', color: '#000', padding: '20px' };


// We call the registerBlockType function to register your block type
// for use in the editor. We need to namespace our blocks 'your-namespace/your-block'
// to avoid collisions between our blocks and others	
registerBlockType( 'cornerstone-block/test-block', {
  title: 'Cornerstone Block 1',
  // icon can be a dash icon or a path to an SVG/PNG
  icon: 'universal-access-alt',
  category: 'layout',
  // Here we define the edit and save methods. Both of these should return a call to 
  // wp.element.createElement with the content we want rendered in our block on edit and save
  edit: function() {
    return el( 'p', { style: blockStyle }, 'This is what appears in the editor.' );
  },

  save: function() {
    return el( 'p', { style: blockStyle }, 'This is what gets saved into post content.' );
  },
} );

Hopefully, most of the comments are enough, but the gist is that we call wp.blocks.registerBlockType and pass it a configuration object that contains save and edit methods, both of which return an element created using wp.element.createElement, which is really a wrapper around React functionality.

One of the things I like already about the block pattern is that I’m not forced to use React. I can use the abstractions provided by the WP JS API instead of dealing with React, at least for most easy things like this. I’m sure I’ll dig into React more over the coming weeks, but I like that I can workaround it for as long as possible.

Enqueue and Register Block in PHP

Now that we have a good sense for the basic JS structure you need to create a block, we need to enqueue the script for the editor and register the block within the core of WP’s PHP. If you’ve done any plugin or theme development in the past, this should look pretty familiar to you already:

function cornerstone_boilerplate_block() {

  // Here we register the script as we would normally, but make
  // sure to pass wp-blocks and wp-element into the dependencies 
  // argument at the end. This tells WP that our script relies on the others  
  wp_register_script(
    'cornerstone-block-1-js',
    plugins_url( 'cornerstone-block-1.js', __FILE__ ),
    array( 'wp-blocks', 'wp-element' )
  );

  // Here we actually register the block with WP, again using our namespacing
  // We also specify the editor script to be used in the Gutenberg interface 
  register_block_type( 'cornerstone-block/cornerstone-block-1', array(
    'editor_script' => 'cornerstone-block-1-js',
  ) );

}
add_action( 'init', 'cornerstone_boilerplate_block' );

Overall, this is still a pretty familiar pattern to most WP devs, and there are even some ways we could make PHP do some rendering here of block content as well, which makes this feel like an easier-to-use shortcode interface. I’ll talk about that in future posts.

What You See Is What You Really Get

One of the largest pushes for a switch to the Gutenberg interface has to do with what can sometimes be a black box of the WP editor interface. Even though TinyMCE is billed as a WYSIWYG editor, in most cases users needed preview the page to see what it would look like styled and rendered.

Gutenberg helps solve some of that problem, by rendering rich HTML directly into the editing interface. For example, this is how our tiny test block is rendered in the editor.

If we look back at our original JS code, we can see that this is the element and content we returned with the edit method attached to our block.

edit: function() {
    return el( 'p', { style: blockStyle }, 'This is what appears in the editor.' );
  },

In a similar fashion, when we update or save the post, the content in the editor switches to whatever element we’ve returned from our save method. This allows us to specify an editing experience for our blocks that is different from how the content is ultimately displayed on the front end.

Our test block uses the following code in our save method:

save: function() {
    return el( 'p', { style: blockStyle }, 'This is what gets saved into post content.' );
  }

And its output looks like this:

Digging into the Gutenberg Editor

Overall, this provides a pretty clear pattern to use when you create blocks. Obviously, we can and will begin to create more sophisticated editing interfaces that require configuration and fuller saved content based on those variables, but this is a good start.

How Blocks Are Saved

Now, you might be asking yourself where some of this configuration data for a custom block gets saved. Most of us have used custom fields to store additional data that we need to construct a custom part of a page, post, or template, but Gutenberg turns that method on its head in a lot of ways. I’m not saying custom fields are going away, or will lose value, but Gutenberg can actually save and retrieve some configuration values from the blocks themselves.

First to understand how this works, lets look at how the content of blocks gets saved into the post content in the database.

Digging into the Gutenberg Editor

Gutenberg uses a system of comments to bracket all of the HTML produced when you save a block. If we look closer at the actual output, there are also classes added to the block content that we can hook into for CSS style.

By structuring the markup in this way, with all of the additional namespacing provided by the comments and the classes, Gutenberg can grab attributes of content from these DOM elements directly, which means that certain types of data can be stored in the blocks themselves as a part of regular post data.

Overall, its a very cool concept, and we’ll look at that in further details in future block examples, but it should make life a bit easier for developers as well. Meaning we can create a block, define its output, then just deal with it like any other post content. No need to do lots of calls to get_post_meta to construct all of the data we need to render a template.

 

 

 

 

The post Digging into the Gutenberg Editor appeared first on Jeff Everhart.

Using AWS for Data Analysis

Using AWS for Data Analysis

I’m not really sure when this happened, but over the last several years, I’ve started to spend a lot of my personal and professional time working on building out data visualization tools and workflows. In a recent meeting, someone referred to us as data scientists, and we’ve had a good running joke ever since. While I appreciate the nod, I’m not sure I’m ready to self-refer to myself as that. As most self-taught people in tech know, it takes awhile to really feel comfortable calling yourself a title that you don’t really think you deserve, having stumbled backwards or perhaps half-blind into your present skill set.

Either way, as someone good with data stuff (SQL, web dev, dashboards, data viz, and Python), I’m increasingly being asked to provide advice as some sort of subject matter expert on the tools in the modern data toolkit.

While this is a lot of fun, it really starts to make me think about my own practices, which are admittedly ever evolving and self-serving. On a recent project, I was asked to advise a professor in the Information Systems department teaching a Business Intelligence class for the Online MBA at VCU. He has a pretty ambitious goal: get MBA students to conduct basic BI/data analysis tasks using tools available in AWS.

Below are just a few of my thoughts on using some of the tools available in AWS.

AWS Athena: Who Needs a Stinking Database

One of the first tools I started looking at was AWS Athena. This is a very cool service that simplifies a lot of things for anyone looking to work with data. In most cases, if you have more than one spreadsheet of data to analyze, or want to do some advanced querying, you will want to load you data into one or more SQL tables using some SQL variant.

Basically, there are a few ways to get data from one source, say a CSV, into a SQL database. Some of the ways I’ve done this in the past involve either loading the CSV files using a database management tool, or script the same thing with Python. As an example, for my IPEDS data analysis, I spent tons of upfront time importing CSV data to create my initial SQL tables. Some of these tables have 200K+ rows, so as you can imagine, that process took some time and was not without error.

While this was clearly the best path, CSV -> SQL, it took an unnecessarily long time considering all I wanted to do was run few queries, then export the resultant data as JSON for use elsewhere. This is where Athena comes in handy.Using AWS for Data Analysis

Athena allows you to upload your data to S3, then create ‘virtual’ databases and tables from that structured data (CSV, TXT, JSON). From there you can use an interface, or I’m assuming an API as well, to run queries against that data directly from S3.

Full blown databases are great at providing read/write access, while also giving you access to more advanced querying tools. But if you are working with a static dataset like I am in the IPEDS project, I’m not really reaping any of the benefits of the database, aside from the SQL querying.

At the same time, AWS’ provisioned database service is really expensive, like almost 2X the cost of a virtual server from the same specs. And the cost of those is calculated in hours, not just the time spent querying or writing data. Athena on the other hand, only charges you for the queries you run, which would make it a more cost effective choice for organizations that have large amounts of data.

Needless to say, I’m going to shift some of my own projects over to using this technology, especially when I’m sure I won’t need to continue to write data to tables. For just loading and transforming the data, it seems like Athena is the way to go.

AWS Quicksight: Why Do I Even Program?

Over the last few years, I’ve built out a number of dashboards manually using some variant of D3 or other charting libraries. However, the more time I spend with data folks, the more and more I question whether or not ‘bespoke’ dashboards written in JavaScript are the way to go.

Technology has a funny way of making hard things trivial over time, and I think we’re about there with modern “business intelligence” tools like Tableau, Power BI, and Quicksight.

Sticking with my theme of AWS for most things, I decided to give Quicksight a try. Since I just wanted to build out some basic examples, I downloaded a quick CSV file of active real estate listings in Richmond, VA from Redfin.

Real estate tends to be a nice domain for test data sets since there are tons of variables to look at, the data sources are plentiful, and it is easy to understand.

Overall, my experience using Quicksight to build out a dashboard was pretty excellent, and I was able to use a nice drag and drop interface to design my visuals.

Using AWS for Data Analysis

At a high level, Quicksight handles data like a boss. It does a great job of inferring the types of data from your source. Without coercion, it picked up the geospatial components of my data set, along with the other numerical and textual data.

Using AWS for Data Analysis

Again, it impressed me with the breadth of available visual models, and the ease with which I could construct them. For most of the visuals, such as this heat map, I’ve built an equivalent chart type somewhere in JavaScript. While I was a bit disappointed in my ability to customize the display of the different charts, I was impressed with how easy there were to create.

Using AWS for Data Analysis

It seems like Quicksight, and I’m sure to a greater extent Tableau, are trying to strike a balance between ease of use and customization. It appears there is only so much I can do to make things look better, but there is only so much visual harm I can do as well.

In the end, I really liked using Quicksight, and it made me take a second to question when a tool like this is a better choice over some sort of web dashboard. However, Quicksight is built around the idea of an ‘account’ with users, and does not appear to have an easy way to publish these visuals to the public web, which for my work is a huge downside. I think this is where Tableau might have an edge with their public gallery or paid hosting.

 

The post Using AWS for Data Analysis appeared first on Jeff Everhart.

Using AWS for Data Analysis

Using AWS for Data Analysis

I’m not really sure when this happened, but over the last several years, I’ve started to spend a lot of my personal and professional time working on building out data visualization tools and workflows. In a recent meeting, someone referred to us as data scientists, and we’ve had a good running joke ever since. While I appreciate the nod, I’m not sure I’m ready to self-refer to myself as that. As most self-taught people in tech know, it takes awhile to really feel comfortable calling yourself a title that you don’t really think you deserve, having stumbled backwards or perhaps half-blind into your present skill set.

Either way, as someone good with data stuff (SQL, web dev, dashboards, data viz, and Python), I’m increasingly being asked to provide advice as some sort of subject matter expert on the tools in the modern data toolkit.

While this is a lot of fun, it really starts to make me think about my own practices, which are admittedly ever evolving and self-serving. On a recent project, I was asked to advise a professor in the Information Systems department teaching a Business Intelligence class for the Online MBA at VCU. He has a pretty ambitious goal: get MBA students to conduct basic BI/data analysis tasks using tools available in AWS.

Below are just a few of my thoughts on using some of the tools available in AWS.

AWS Athena: Who Needs a Stinking Database

One of the first tools I started looking at was AWS Athena. This is a very cool service that simplifies a lot of things for anyone looking to work with data. In most cases, if you have more than one spreadsheet of data to analyze, or want to do some advanced querying, you will want to load you data into one or more SQL tables using some SQL variant.

Basically, there are a few ways to get data from one source, say a CSV, into a SQL database. Some of the ways I’ve done this in the past involve either loading the CSV files using a database management tool, or script the same thing with Python. As an example, for my IPEDS data analysis, I spent tons of upfront time importing CSV data to create my initial SQL tables. Some of these tables have 200K+ rows, so as you can imagine, that process took some time and was not without error.

While this was clearly the best path, CSV -> SQL, it took an unnecessarily long time considering all I wanted to do was run few queries, then export the resultant data as JSON for use elsewhere. This is where Athena comes in handy.Using AWS for Data Analysis

Athena allows you to upload your data to S3, then create ‘virtual’ databases and tables from that structured data (CSV, TXT, JSON). From there you can use an interface, or I’m assuming an API as well, to run queries against that data directly from S3.

Full blown databases are great at providing read/write access, while also giving you access to more advanced querying tools. But if you are working with a static dataset like I am in the IPEDS project, I’m not really reaping any of the benefits of the database, aside from the SQL querying.

At the same time, AWS’ provisioned database service is really expensive, like almost 2X the cost of a virtual server from the same specs. And the cost of those is calculated in hours, not just the time spent querying or writing data. Athena on the other hand, only charges you for the queries you run, which would make it a more cost effective choice for organizations that have large amounts of data.

Needless to say, I’m going to shift some of my own projects over to using this technology, especially when I’m sure I won’t need to continue to write data to tables. For just loading and transforming the data, it seems like Athena is the way to go.

AWS Quicksight: Why Do I Even Program?

Over the last few years, I’ve built out a number of dashboards manually using some variant of D3 or other charting libraries. However, the more time I spend with data folks, the more and more I question whether or not ‘bespoke’ dashboards written in JavaScript are the way to go.

Technology has a funny way of making hard things trivial over time, and I think we’re about there with modern “business intelligence” tools like Tableau, Power BI, and Quicksight.

Sticking with my theme of AWS for most things, I decided to give Quicksight a try. Since I just wanted to build out some basic examples, I downloaded a quick CSV file of active real estate listings in Richmond, VA from Redfin.

Real estate tends to be a nice domain for test data sets since there are tons of variables to look at, the data sources are plentiful, and it is easy to understand.

Overall, my experience using Quicksight to build out a dashboard was pretty excellent, and I was able to use a nice drag and drop interface to design my visuals.

Using AWS for Data Analysis

At a high level, Quicksight handles data like a boss. It does a great job of inferring the types of data from your source. Without coercion, it picked up the geospatial components of my data set, along with the other numerical and textual data.

Using AWS for Data Analysis

Again, it impressed me with the breadth of available visual models, and the ease with which I could construct them. For most of the visuals, such as this heat map, I’ve built an equivalent chart type somewhere in JavaScript. While I was a bit disappointed in my ability to customize the display of the different charts, I was impressed with how easy there were to create.

Using AWS for Data Analysis

It seems like Quicksight, and I’m sure to a greater extent Tableau, are trying to strike a balance between ease of use and customization. It appears there is only so much I can do to make things look better, but there is only so much visual harm I can do as well.

In the end, I really liked using Quicksight, and it made me take a second to question when a tool like this is a better choice over some sort of web dashboard. However, Quicksight is built around the idea of an ‘account’ with users, and does not appear to have an easy way to publish these visuals to the public web, which for my work is a huge downside. I think this is where Tableau might have an edge with their public gallery or paid hosting.

 

The post Using AWS for Data Analysis appeared first on Jeff Everhart.

Contact us

Academic Learning Transformation Lab - ALT Lab
1000 Floyd Ave, Suite 4102 | Richmond, Virginia 23284
altlab@vcu.edu | 804-827-5181

Last updated: April 11, 2018

Virginia Commonwealth University

Privacy | Accessibility | Webmaster