OmniFocus Progress Chart Part 1: Extract & Save Data

Measured progress motivates me. I want to see how many things are in my OmniFocus library and if it’s trending larger or smaller. My desktop has been home to a list of today’s completed OmniFocus tasks for a long time, and now it’s home to a pair of charts: remaining items over the last 30 days, and items completed per day for the last 30 days.

The charts are created in 3 steps, and I am breaking these up into two posts:

  1. Extract a snapshot of the status of the database every day, (this post).
  2. Use that snapshot to create a text-based chart, (next post).
  3. Show the chart on my desktop, (next post).

Steps 1 & 2 are handled by a Javascript for Automation application written in Script Editor. Step 3 is handled by GeekTool.

Extracting & Saving the Database Status

There are a few ways to build a program that interacts with other programs on a Mac. For the completed task list I used Automator, but I don’t like editing Java- or Apple- script in the tiny Automator windows, so I tried out Script Editor. Script Editor lets us create an app bundle with somewhat modularized code, but isn’t the greatest IDE and crashes more than it should. Save your work often.

Applescript is foreign to me so I opted for Javascript for Automation, aka JXA, Apple’s, (maybe abandoned?), attempt to get a Javascript version of Applescript running. It works ok, but documentation of how it actually works, and what parts of modern Javascript are supported, is hard to find.

To extract the data we can use OmniFocus’s flattenedTasks list, which gets a flattened list of all tasks in the database, then we can filter that list of tasks by status, creation date, completion date, and so on:

// Get the "document" that we need to work with.
const ofdoc = Application('OmniFocus').defaultDocument;

// Get a list of all tasks in the DB.
const tasks = ofdoc.flattenedTasks;
// now tasks.length is the total number of tasks in the DB
// (this will change a lot when you archive old tasks)

// Filter using the .whose method.
const remainingTasks = tasks.whose({
      effectivelyCompleted: false,
      effectivelyDropped: false
});

// ... etc. for completed tasks, and tasks completed/added/dropped in the past day.

Filtering with whose is pretty slow so this can’t be run every few seconds, (it takes several seconds to run), but for now I’m only updating daily so it’s fine if it’s slow.

I tried iterating over the tasks and checking the effectivelyCompleted property on each one, thinking it would be faster, but effectivelyCompleted has to send a message to OmniFocus to get a response, and doing that for my entire library is much slower than a single .whose() call.

Once all the tasks are filtered the current stats put into a Javascript object, which is .push()‘ed onto the end of a Javascript array of all the stats and stored on disk as a JSON file.

Modularizing the code was a hurdle.

I tried using module exports & imports but they don’t work. It seems like there is an import() function available but I can’t find what it does. It’s hard to see what Javascript/ECMAScript features are supported in JXA. Some documentation says that it uses the same Javascript engine as Safari, but if that was true when it was released it doesn’t appear to be true now.

Even without exports & imports the Release Notes say we can import libraries into our script like so:

// Imports from StatsProcessor.scpt - supposedly. Also, where is StatsProcessor.scpt?
const StatsProcessor = Library("StatsProcessor");

The 10.11 Release Notes say that we can put our library scripts into the ‘Contents/Library/Script Libraries’ directory of the App we’re making. I couldn’t find a way to do this, or even see the Script Libraries folder in Script Editor, but once I moved everything around with finder the library was recognized and the import worked. But this code crashed:

const StatsProcessor = Library("StatsProcessor");
StatsProcessor.anyPublicMethod();

Not only did it crash, but it made Script Editor crash as well. Calling any method on an imported library caused a crash. Apparently we need to use a “compiled” script – a .scptd file – for libraries. This isn’t documented anywhere that I can find, and there seems to be no way to change between a .scpt and .scptd file, at least not using a GUI.

There’s also no documentation for that a Library actually is and what’s available when it’s imported. From what I can see any global function declared in the imported file is available as a method on the imported object.

Once the modularization was working it was relatively straightforward to figure out where to store the data, (answer: In ~/Library/Application Support/OmniFocusStats), and save it as desired.

Next up: Using the data to make & display a chart.

Paste!

Once in a while I run into websites that block pasting in password, (or confirm password), fields. Like most bad ideas it doesn’t happen often but it happens just enough that I no longer want to manually type out the javascript to re-enable pasting.

So I made a bookmarklet. Here it is:

Paste!

Drag it to your bookmark toolbar to install it in your browser. When you find a page blocks paste click the bookmark and paste should be restored.

How it works

When clicked a bit of Javascript executes that adds an event listener to the capturing phase of the paste event and stops any further propagation of the event. Because we usually react to the bubble phase of DOM events, and the capture phase comes first, the bookmarklet’s event listener captures the event, lets the default action happen, and prevents all other JS from listening to it, preventing the web page from blocking your carefully-pasted text. The actual code is a shortened version of the function at How to Enable Pasting Text on Sites that Block It.

Changing the Auto-Suggest Behaviour in WooCommerce

A request came in this week from a client with a WooCommerce store:

I just placed an order through our website and had an issue with the state. The customer lives in Oregon, so I typed “O” hoping Oregon would auto populate, but it didn’t. I was stuck having to scroll through all 50 states to find Oregon.

An animated Gif showing WooCommerce's default autosuggest behaviour on select boxes.
Confusing for most people.

I tried it, and it’s true. Typing “o” doesn’t narrow the list down much, and the way it does narrow it down doesn’t make much sense in this context: The autosuggest on the State field matches the search phrase anywhere in one of the options, so typing “o” gets you a list of all states with the letter “o” in the state name, ordered alphabetically by state name. Go ahead and try it out in a WooCommerce demo shop.

Out of the box WooCommerce uses SelectWoo, which is essentially Select2 with more accessibility. Select2’s default behaviour of matching the typed text to any part of an option makes sense in many places, especially when searching a store or for categories, but for selecting states, (and countries, although slightly less), it’s confusing.

Nothing’s filterable, but we can set a default

Select2 lets us set the callback function used for matching, so we can customize the matching behaviour, but there’s a problem, the settings for SelectWoo are set in WooCommerce’s country-select.js file, and there’s no way to filter them, so we can’t change the matching behaviour on initialization. It turns out that once a Select2/SelectWoo instance is initialized there’s no way to change the settings either.

When I got to here I thought we were out of luck – either I’d be creating a child theme and re-implementing the checkout, or we were going to live with SelectWoo’s default filtering behaviour. Luckily, I found one way to change the matching callback: It’s possible to set the default SelectWoo options callback before any instances are initialized. So that’s what we’re going to do.

Server-Side

First we need to add a bit of Javascript to our checkout page, so we enqueue a small script that’s dependent on the selectWoo script.


namespace JB\WooAutosuggest;



function enqueue_frontend_scripts() {
    
    if( function_exists('is_checkout') && is_checkout() ) {

        wp_enqueue_script( 'jb-was-checkout-autosuggest', plugins_url( '../assets/checkout-autosuggest.js', __FILE__), array('selectWoo') );

    }
}

add_action( 'wp_enqueue_scripts', __NAMESPACE__ . '\enqueue_frontend_scripts', 20 );

And that’s all the PHP we need! Because our solution changes the default matching behaviour we are careful to only enqueue this script on the Checkout page using WooCommerce’s is_checkout() function.

Client-Side

The enqueued Javascript file is pretty simple as well:

(function($) {
    // Based on custom matcher here: https://select2.org/searching#customizing-how-results-are-matched
    function start_matcher(params, data ) {
        
        // If there are no search terms, return all of the data
        if ($.trim(params.term) === '') {
          return data;
        }

        // Do not display the item if there is no 'text' property
        if (typeof data.text === 'undefined') {
          return null;
        }

        // `params.term` should be the term that is used for searching
        // `data.text` is the text that is displayed for the data object
        // Make sure to compare in the same case! 
        if ( data.text.toLowerCase().indexOf(params.term.toLowerCase() ) === 0 ) {
            return data;
        }
        // Return `null` if the term should not be displayed
        return null;
    }
    $.fn.select2.defaults.set('matcher', start_matcher );
})(jQuery);

There’s a closure so we don’t pollute the global namespace, then we define a new matcher function that matches the beginning of the phrase, and set it as the default matcher. This code runs as soon as it’s ready – it doesn’t wait for a DOMReady event, because by the time DOMReady fires the SelectWoo boxes have probably been initialized already.

Conclusion

I wish we didn’t have to set the defaults for the whole page, but that’s what we’re stuck with at the moment. We could apply different matching patterns to different pages by adding a few more conditions to the PHP block, and a bit more Javascript. This works to make the checkout page work as expected, though.

All the code is up on Github as a WordPress plugin. If it’s useful let me know!

An animated Gif showing WooCommerce's autosuggest matching the beginning of each option.
This makes more sense!

Vanilla Speed

Me: I wonder how to make this cool new HTML5 feature interact with Javascript properly.

Googles…

Stack Overflow Answer #1: Use this great jQuery function!

Stack Overflow Answer #2: Underscore.js is better, us it! You only need 1/2 a character of code.

Smacks desk in frustration.

I’m working on a major overhaul of an existing website that focuses on speed and a great mobile experience that has never used Javascript libraries, (it was originally written in 2007 before they were a thing), and we’re not changing that. For this website a library is a bunch of extra code to download, execute, and take up memory, which adds extra function calls between the code I write and the things that happen in the browser – more CPU cycles, more battery drain, more waiting, and no benefit to our users and customers.

If we were using jQuery I wouldn’t have spent some time looking up the differences in event creation between browsers, and I might not have spent the morning figuring out why a datalist wasn’t appearing properly in Firefox, or maybe I would have had to do that bit of debugging anyway. One thing that doesn’t help me move the project along at all is other people’s over-reliance on Javascript libraries that is so prevalent. In 2007 when I had a problem I could Google it, or post to a forum, and find an answer, (assuming it had been done before), today I experience the sequence of events at the top of this post several times every day. This is why the trio of blog posts last week, (Marco Arment’s post about PPK’s post about John Gruber’s take on Facebook Instant), have really resonated with me. PPK is right, we need to stop relying on libraries, whether it’s jQuery, underscore, or whatever. There are times when a library is the right tool, but with modern Javascript APIs, HTML5, and CSS3, and good browser support for all but the absolute bleeding-edge.

To use a Javascript library is to externalize the cost of development onto your customers and users.

Despite my frustration with the “use some library” responses on Stack Overflow questions using plain vanilla JavaScript has turned out to be an enjoyable learning experience. I’ve been able to make everything work, (so far), with way less code than was needed in 2007. Of course, if your browser can’t handle the project’s Javascript requirements the JS is not loaded at all and you get the plain HTML experience, which isn’t so bad either.

For more on choosing not to burden your users and customers with unneeded slowness, see the ALA article Choosing Vanilla Javascript, which is 15 months old now, and things have just gotten better since it was published!

Really Purging a YouTube Iframe in Internet Explorer

I’ve been working on a website where we display YouTube videos in a modal overlay. This seems simple: absolutely position a div and centre YouTube’s iframe embed within it. When the user tires of the video remove the iframe and hide or remove the div. It works great, then I fired up Internet Explorer for testing.

In IE this is what appears in the Javascript console,  after the iframe is removed:

Errors in the IE Javascript Error Console.
These errors appear after html5player.js shouldn’t exist anymore.

With the iframe gone html5player.js and its code shouldn’t  exist anymore and therefore should not throw errors. It turns out that when an iframe is removed from a web page IE doesn’t completely purge the contents of the iframe from memory right away, if ever.

How do we fix it?

The errors occurred when using jQuery to set the innerHTML of the containing div to an empty string, so I started trying to explicitly remove the iframe’s DOM node using pure Javascript. Element.removeChild() the node instead of destroying it, so that didn’t kill the errors. The docs for IE’s removeNode() don’t specify if the node is held in memory, but the errors continued with it as well.

Some Googling turned up, (I forget where), that setting the iframe’s src attribute to an empty string might help. This makes sense because it’s the equivalent of navigating the iframe to a new page, which should unload any Javascript. Trying to set src=” worked, most of the time. When it didn’t work IE went crazy and became unusable.

In the end I took the approach of trying to wipe out the contents of the iframe as thoroughly as possible, first by changing the iframe’s src to an empty string, then by accessing the iframe’s DOM with Javascript and setting its document element to null. This did the trick. Here’s the function:

// note: modal is the div that contains the YouTube iframe
// jquery is used for some selecting, but the guts of the removal are pure Javascript

function purgeVideo() {
    // purge video must be called while the video is still displayed on the page, 
    // so doing a jQuery fadeOut(purgeVideo) may throw errors, (I think because 
    // fadeOut sets display:none before calling its callback).

    if( modal ) {
        // get the iframe, declare variables
        var ifr = $('iframe', modal), rifr;
        
        // if the iframe is found get the underlying DOM node from jQuery
        if ( ifr.length > 0 ) {
            rifr = ifr.get(0);
        }

        // make extra sure the DOM node exists before starting to work on it.
        if( rifr ) {
            // set src to '' to start removing YouTube and get around cross-domain access restrictions
            rifr.src = '';

            // now set the document element of the iframe to null, (I think this is IE-specific)
            if ( rifr.documentWindow ) {
                rifr.documentWindow.document = null;
                rifr.documentWindow = null;
            }

            // delete the iframe, (also removes it from DOM).
            delete rifr;
        }

        // this won't throw an error, even if the iframe is gone, (we're back to jQuery here).
        ifr.remove();

        // remove anything else, (titles, etc), from the modal overlay
        modal.html('');
    }

}