Category: Tools

Automating the Fracturing of a Git Repository

I have a git repo that’s become a monster. It’s got at least two WordPress themes, a handful of custom WordPress plugins, some .htaccess files, an artwork directory, and more. Back in the dark times, when it was a Subversion repository this sort of made sense, in that I didn’t have to set up a new repo on the server for each component in the project. But I haven’t touched SVN in a long time, and making new repositories in Git is easy. We’re doing some major work on this client’s sites this fall, so it’s time to break up the giant repository into several smaller ones.

The monolith repository is being split into seventeen smaller components. One of these components is an “artwork” directory, the ~600MB history of which will bloat the git history of all sixteen other repositories if we don’t purge it properly. The goals of this script are:

  1. Avoid typing the same set of commands seventeen times.
  2. Keep the appropriate git history for each of the seventeen components.
  3. Purge any unrelated git history so each component only has its own history in its .git directory.

Github has a help page on splitting a subfolder into its own repository, and another on removing files from a repository’s history, (it’s meant for sensitive data, but works for all data). In theory we only really need the first link, but in testing the entire history of the monolith remained when only following the instructions in the first link, but with the purge and garbage collection commands from the second link the git history is down to an appropriate size.

The script is relatively simple, but relies a bit on directory structure. It should be in the same directory as the monolith repo, (the script doesn’t go into the monolith, it goes in the same containing folder). There are two places to change the script:

  1. Put the directories for extraction into the repos array, (starting on line 5). These paths are relative to the root of the repository.
  2. Put the path of a copy of the monolith repo for use as a source in the master variable, (line 16).

Once that’s done, running ./export-directory-repos.sh should export the repositories.

Here’s the gist:


#!/bin/bash
# Enter the paths in the main repo to the subdirectories you want to extract
# Separate paths with spaces or newlines
repos=(
artwork
checklists
code/comingsoon
code/utility
)
echo "Total repos to export : ${#repos[*]}"
# Put the directory of the main repo here.
master=monolith-repo-master
for repo in "${repos[@]}"
do
subdir=$(basename $repo)
echo "${repo} => ${subdir}"
cp -R $master $subdir
cd $subdir
git filter-branch –prune-empty –subdirectory-filter $repo master
git for-each-ref –format='delete %(refname)' refs/original | git update-ref –stdin
git reflog expire –expire=now –all
git gc –prune=now
cd ../
done

view raw

export-directory-repos.sh

hosted with ❤ by GitHub

 

 

Flush Opcache with Varnish: a WordPress Plugin Announcement

Flush Opcache with Varnish is here to rescue you from the constant annoyance of flushing yet another cache!

If, like me, you use PHP’s Opcache to speed up a site, and you have a Varnish cache, and maybe some other server-side caches, you probably want to flush the opcache, Varnish cache, and any other server-side caches at the same time after changing the PHP files on your server, (like, when you update WordPress, plugins, or themes). I don’t want to click a button for each cache type, or worse, have to log in to the command-line for a sudo service restart varnish after doing updates, especially if I’m doing repeated updates. Flush Opcache with Varnish hooks into Mika Epstein’s excellent Varnish HTTP Cache plugin and flushes the PHP Opcache and the WP Super Cache cache, (if you use WP Super Cache), every time you manually flush the Varnish cache, turning the Varnish HTTP Cache “Clear Cache” buttons into a three-for-one deal.

If you also use the plugin WP Opcache to manage your opcache then WP Opcache will be called to do the actual Opcache flushing so you can take advantage of its automatic rebuilding of the opcache.

I’m already using Flush Opcache with Varnish in production and it’s saving me time every time I update a theme or plugin. You can get it in the WordPress.org plugin directory.

A PHP Version Switcher for the AMP stack on OS X

A friend is thinking about buying MAMP Pro so he can test on different versions of PHP. Since I just set up version-switching myself, for free, here’s how I did it. Hopefully it’ll save someone a bit of money or frustration.

This is happening on a Mac, with the latest version of OS X, (Currently 10.11.3 El Capitan), using the pre-installed version of Apache, and Homebrew as a package manager.

Step 1: Install more than one version of PHP on your system

Using Homebrew you can install a version of PHP, then “unlink” it so it’s not currently being used:

brew install php7
brew unlink php7

If you want to be able to switch PHP versions on the command-line there’s a tool called PHP Version that does the heavy lifting for you, and with Homebrew it’s easy to install.

Step 2: Make it easy to change the PHP version Apache is using

First, stop the main Apache config, (/etc/apache2/httpd.conf), file from loading PHP. Find any line that mentions PHP modules and comment it out, (put a # at the beginning of the line). For example:

#LoadModule php5_module libexec/apache2/libphp5.so
#LoadModule php5_module /usr/local/opt/php56/libexec/apache2/libphp5.so

Now we need an Apache config file for each PHP version that you want to run. I’ve put these files in /etc/apache2/other/ and used the file extension “.conffile” to prevent Apache from auto-loading them:

A screenshot of /etc/apache2/conf/other

Each of your .conffile files needs to have the location of the PHP library, and any extra PHP info. This is the contents of my php-7.conffile:

LoadModule php7_module /usr/local/opt/php70/libexec/apache2/libphp7.so
<IfModule php7_module>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
</IfModule>
<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

php-5.conffile is very similar, but has the path of the PHP 5 .so file and the IfModule conditional checks for php5_module.

If you have any existing php*.conf files in /etc/apache2/other/ they can be blank.

Next we need to make an easy way to tell Apache to use one of the .conffiles, which I’ve  done by creating a symbolic link to the one I want to use. Since I don’t like typing out ln commands I’ve created a script, which I call php-v.sh.

#!/bin/bash
if [ $1 = 7 ]; then
PHP_VERSION=7
else
PHP_VERSION=5
fi
`sudo ln -Fs /etc/apache2/other/php-$PHP_VERSION.conffile /etc/apache2/other/php.conf`
`sudo apachectl graceful`
echo "PHP version switched to $PHP_VERSION"

This script symlinks /etc/apache2/other/php.conf to the .conffile that I want to use, (using the -F switch to overwrite the old symlink), and restarts Apache. It needs sudo so it’ll ask for your password.

Finally, it’s nice to have php-v.sh available from everywhere, so I’ve created a symlink from ~/bin/php-v, which is in my PATH, to the actual script:


ln -s /Users/John/Developer/Scripts/php-v.sh /Users/John/bin/php-v

And now I can switch PHP versions at will:A screenshot of my terminal, having just switched PHP versions quickly & easily!

Show the OmniFocus Tasks You Did Today on your Desktop

This morning Ken Case retweeted this:

Using Automator, not TextExpander

I’ve wanted a solution to show me what I accomplished in a day for a while, but I don’t use TextExpander. Luckily Colter Reed’s script to log today’s completed OmniFocus tasks is Javascript for Automation, which can be run with the built-in OSX app, Automator, so I built an Automator workflow that runs the script, (I just pasted Colter Reed’s script into an Automator “Run Javascript” action, then made one tweak when I ran into a small problem). If you’ve never used Automator before don’t worry – I hadn’t either! Here’s my Automator workflow file, (it’ll need to be unzipped). It returns a bunch of text, which we’ll deal with next.

Put the Results on your Desktop with GeekTool

GeekTool can take text from a shell script and display it on your desktop, and you can set it to refresh that text however often you want. I created a new shell Geeklet with GeekTool with the following script:

Automator Runner ~/Developer/Scripts/omnifocus-completed.workflow | sed -e 's/^"//' -e 's/"$//' -e 's/\\"/"/g'

For some reason the Automator Runner returns the text with quotation marks around it, so the parts after the pipe remove the quotation marks, (using instructions from Stackoverflow).

When choosing your refresh interval keep in mind that the script grabs focus from OmniFocus when it runs, which will be annoying if you are working in OmniFocus and you have a short refresh interval. I also don’t know how much battery this will eat for those of us on laptops.

With some font & colour tweaking, my desktop now looks like this:

omnifocus-completed-desktop-screensho

Update (October 9, 2015): This morning a bunch of tasks I did last night, but before midnight, were still on my desktop, so I went debugging. It looks like OmniFocus, (at least my version, which is a pre-release test version), is exposing the completedDate value to Javascript as a UTC date/time, but with a Timezone offset set, so things done in the evening might remain on the list of “today’s” completed tasks, (this might reverse on the other side of the world, with things you do in the morning not appearing). To fix this I’ve modified the startOfDay() function to take the Timezone Offset into account. I assume this is a bug and will be temporary, so I’m not updating the downloadable Automator action above. Here’s the updated startOfDay() function:


function startOfDay() {
// The day started at midnight this morning
    var d = new Date(),
    hours = 0,
    minutes = 0;
    
    if( d.getTimezoneOffset() !== 0 ) {
        hours = Math.round( d.getTimezoneOffset() / 60 );
        minutes = d.getTimezoneOffset() % 60;
    }
    
    d.setHours( hours );
    d.setMinutes( minutes );
    d.setSeconds( 0 );
    
    return d;
}

Update (November 3, 2015): I used quotation marks in a task name, and they came out escaped on my desktop. I’ve updated the command line for the geeklet to strip the slashes from before double quotation marks;

Vanilla Speed

Me: I wonder how to make this cool new HTML5 feature interact with Javascript properly.

Googles…

Stack Overflow Answer #1: Use this great jQuery function!

Stack Overflow Answer #2: Underscore.js is better, us it! You only need 1/2 a character of code.

Smacks desk in frustration.

I’m working on a major overhaul of an existing website that focuses on speed and a great mobile experience that has never used Javascript libraries, (it was originally written in 2007 before they were a thing), and we’re not changing that. For this website a library is a bunch of extra code to download, execute, and take up memory, which adds extra function calls between the code I write and the things that happen in the browser – more CPU cycles, more battery drain, more waiting, and no benefit to our users and customers.

If we were using jQuery I wouldn’t have spent some time looking up the differences in event creation between browsers, and I might not have spent the morning figuring out why a datalist wasn’t appearing properly in Firefox, or maybe I would have had to do that bit of debugging anyway. One thing that doesn’t help me move the project along at all is other people’s over-reliance on Javascript libraries that is so prevalent. In 2007 when I had a problem I could Google it, or post to a forum, and find an answer, (assuming it had been done before), today I experience the sequence of events at the top of this post several times every day. This is why the trio of blog posts last week, (Marco Arment’s post about PPK’s post about John Gruber’s take on Facebook Instant), have really resonated with me. PPK is right, we need to stop relying on libraries, whether it’s jQuery, underscore, or whatever. There are times when a library is the right tool, but with modern Javascript APIs, HTML5, and CSS3, and good browser support for all but the absolute bleeding-edge.

To use a Javascript library is to externalize the cost of development onto your customers and users.

Despite my frustration with the “use some library” responses on Stack Overflow questions using plain vanilla JavaScript has turned out to be an enjoyable learning experience. I’ve been able to make everything work, (so far), with way less code than was needed in 2007. Of course, if your browser can’t handle the project’s Javascript requirements the JS is not loaded at all and you get the plain HTML experience, which isn’t so bad either.

For more on choosing not to burden your users and customers with unneeded slowness, see the ALA article Choosing Vanilla Javascript, which is 15 months old now, and things have just gotten better since it was published!