Preview Laravel Password Reset and Verification E-mails in a Browser

I’m building something on Laravel, using the built-in Auth with 5.7’s new E-mail Verification feature. Today’s task was to tweak the design of E-mail notifications a bit, so I wanted to preview them in a browser. I thought it would be easy, but the built-in E-mails use the Notification system, bypassing Mailable objects. Some Googling didn’t help much, but combining the Notifications documentation and this forum post got let me preview the messages. Hopefully this post can help someone else preview their messages much more quickly than I did today.

The concept is similar to the “Previewing Mailables” documentation: define a route to view your preview on, and return something that will render the message. The trick is that the MailMessage objects that the Notifications’ toMail() methods return don’t have a render() method like Mailable class does, so we need to find something else. Since the built-in Auth uses Markdown mail we can use the Markdown class. A Notifiable object, (the User object for the user who will get the notification), and a Notification are also needed.

For this to work you need to have the right Notifications published to your resources/views/vendor directory. This is done with the Artisan command php artisan vendor:publish --tag=laravel-notifications.

And set up the preview routes:


Route::get('/reset-notification', function () {

    // Get a user for demo purposes
    $user = App\User::find(1);

    // Make a Reset Notification object, (subclass of `Notification`)
    $resetNotification = new \Illuminate\Auth\Notifications\ResetPassword( 'some-random-string-this-will-be-much-longer-in-real-life' );

    // get the `MailMessage` object
    $mailMessage = $resetNotification->toMail($user);

    // get the `Markdown` object
    $markdown = new Illuminate\Mail\Markdown(view(), config('mail.markdown'));

    // Render the vendor.notifications.email view with data from the `MailMessage` object
    return $markdown->render('vendor.notifications.email', $mailMessage->toArray());
});


Route::get( '/verify-notification', function () {

    // Get a user for demo purposes
    $user = App\User::find(1);

    // Make a Verify Notification object, (subclass of `Notification`)
    $verifyNotification = new Illuminate\Auth\Notifications\VerifyEmail();

    // get the `MailMessage` object
    $mailMessage = $verifyNotification->toMail($user);

    // get the `Markdown` object
    $markdown = new Illuminate\Mail\Markdown(view(), config('mail.markdown'));

    // Render the vendor.notifications.email view with data from the `MailMessage` object
    return $markdown->render('vendor.notifications.email', $mailMessage->toArray());
});

Once the Notification is published and the routes created it is possible to preview the Password Reset and E-mail Verification E-mails in the browser.

Changing the Auto-Suggest Behaviour in WooCommerce

A request came in this week from a client with a WooCommerce store:

I just placed an order through our website and had an issue with the state. The customer lives in Oregon, so I typed “O” hoping Oregon would auto populate, but it didn’t. I was stuck having to scroll through all 50 states to find Oregon.

An animated Gif showing WooCommerce's default autosuggest behaviour on select boxes.
Confusing for most people.

I tried it, and it’s true. Typing “o” doesn’t narrow the list down much, and the way it does narrow it down doesn’t make much sense in this context: The autosuggest on the State field matches the search phrase anywhere in one of the options, so typing “o” gets you a list of all states with the letter “o” in the state name, ordered alphabetically by state name. Go ahead and try it out in a WooCommerce demo shop.

Out of the box WooCommerce uses SelectWoo, which is essentially Select2 with more accessibility. Select2’s default behaviour of matching the typed text to any part of an option makes sense in many places, especially when searching a store or for categories, but for selecting states, (and countries, although slightly less), it’s confusing.

Nothing’s filterable, but we can set a default

Select2 lets us set the callback function used for matching, so we can customize the matching behaviour, but there’s a problem, the settings for SelectWoo are set in WooCommerce’s country-select.js file, and there’s no way to filter them, so we can’t change the matching behaviour on initialization. It turns out that once a Select2/SelectWoo instance is initialized there’s no way to change the settings either.

When I got to here I thought we were out of luck – either I’d be creating a child theme and re-implementing the checkout, or we were going to live with SelectWoo’s default filtering behaviour. Luckily, I found one way to change the matching callback: It’s possible to set the default SelectWoo options callback before any instances are initialized. So that’s what we’re going to do.

Server-Side

First we need to add a bit of Javascript to our checkout page, so we enqueue a small script that’s dependent on the selectWoo script.


namespace JB\WooAutosuggest;



function enqueue_frontend_scripts() {
    
    if( function_exists('is_checkout') && is_checkout() ) {

        wp_enqueue_script( 'jb-was-checkout-autosuggest', plugins_url( '../assets/checkout-autosuggest.js', __FILE__), array('selectWoo') );

    }
}

add_action( 'wp_enqueue_scripts', __NAMESPACE__ . '\enqueue_frontend_scripts', 20 );

And that’s all the PHP we need! Because our solution changes the default matching behaviour we are careful to only enqueue this script on the Checkout page using WooCommerce’s is_checkout() function.

Client-Side

The enqueued Javascript file is pretty simple as well:

(function($) {
    // Based on custom matcher here: https://select2.org/searching#customizing-how-results-are-matched
    function start_matcher(params, data ) {
        
        // If there are no search terms, return all of the data
        if ($.trim(params.term) === '') {
          return data;
        }

        // Do not display the item if there is no 'text' property
        if (typeof data.text === 'undefined') {
          return null;
        }

        // `params.term` should be the term that is used for searching
        // `data.text` is the text that is displayed for the data object
        // Make sure to compare in the same case! 
        if ( data.text.toLowerCase().indexOf(params.term.toLowerCase() ) === 0 ) {
            return data;
        }
        // Return `null` if the term should not be displayed
        return null;
    }
    $.fn.select2.defaults.set('matcher', start_matcher );
})(jQuery);

There’s a closure so we don’t pollute the global namespace, then we define a new matcher function that matches the beginning of the phrase, and set it as the default matcher. This code runs as soon as it’s ready – it doesn’t wait for a DOMReady event, because by the time DOMReady fires the SelectWoo boxes have probably been initialized already.

Conclusion

I wish we didn’t have to set the defaults for the whole page, but that’s what we’re stuck with at the moment. We could apply different matching patterns to different pages by adding a few more conditions to the PHP block, and a bit more Javascript. This works to make the checkout page work as expected, though.

All the code is up on Github as a WordPress plugin. If it’s useful let me know!

An animated Gif showing WooCommerce's autosuggest matching the beginning of each option.
This makes more sense!

Adding Custom Fields to WooCommerce Products in 2018

Screenshot showing the Shipping section of WooCommerce's Product Data box.
Look, there’s a Shipping section!

This week a client asked to add estimated ship times to the single-product pages in WooCommerce. Ship times are sometimes different for different products, so adding it as a custom field to each product made sense. As a bonus, there’s a “Shipping” tab in the Product Data metabox that the field can go in. Some Googling led me to Tom McFarlin‘s “Adding Custom Fields to Simple Products with WooCommerce” tutorial on tuts+, which is pretty thorough, but out of date – the actions he’s are no longer in WooCommerce, so it’s time for a new version of that tutorial, my-style. There are three main tasks to accomplish: Add the field to the WordPress back-end, save whatever is in the field, and output the data on the Single Product page. We’ll look at each task separately. The completed plugin is up on GitHub if you want to follow along.

Add the Custom Field to WooCommerce in the WordPress Back-End

WooCommerce provides an incredible number of hooks and filters. We’re going to us woocommerce_product_options_shipping hook, because it runs in the shipping block, but is an equivalent hook for each Product Data tab. Look at the files in the woocommerce/includes/admin/meta-boxes/views directory to understand how the Product Data, (and all the WooCommerce Metaboxes), are created, and what hooks are available.

Now we know what hook we need, it’s time for some code:


function print_admin_field() {
    /**
     * @var \WC_Product
     * @see https://docs.woocommerce.com/wc-apidocs/source-class-WC_Meta_Box_Product_Data.html#42
     */
    global $product_object;

    if( $product_object->get_type() == 'simple' ) {
        $field = [
            'id'            => 'sif_ships_in',
            'label'         => __('Ships In', 'jb-sif'),
            'placeholder'   => __('example: 1-2 days', 'jb-sif'),
            'value'         => get_post_meta( $product_object->get_id(), '_sif_ships_in', true ), // Pre-fill any values that have already been saved.
            'description'   => __( 'Add a "Ships in X" message below the Product Meta on the single-product page.', 'jb-sif'),
            'desc_tip'      => true
        ];

        \woocommerce_wp_text_input( $field );
    }
}

add_action( 'woocommerce_product_options_shipping', 'JB\SIF\print_admin_field');

We make sure we have the $product_object variable available, (it’s set in the WC_Meta_Box_Product_Data class, linked with the @see directive), then we make sure we’re adding the field to a “Simple” product type. Then, instead of writing a bunch of HTML, we set some info in an array, and pass it to the woocommerce_wp_text_input() function, (with a backslash, because the plugin is namespaced). WooCommerce has several of these helper fields available that create a form fields and their associated markup so they fit nicely with the rest of the WooCommerce admin.

Notice we’re already getting a value with get_post_meta(). That’s because the same code gets used every time the field is displayed – so if there’s a value in the database we want to display it. This is all we need to properly display a field in the Shipping section of the Product Data. We could add a name attribute to the $field variable, but it defaults to the value of id, so I left it out. Setting desc_tip to true puts the description in the tooltip that appears when someone hovers the circle with the question mark beside the field.

The last line of the code block above hooks our function into the woocommerce_product_options_shipping action. Because the plugin has its own namespace we need to include the full name of the function in the add_action() call.

A screenshot of the new "Ships In" field in the WooCommerce admin.
Our new “Ships In” field, looking great.

Save Whatever is Entered in the Field

Again, a game of hooks figuring out how to save the data. Tom’s tutorial from last year used a woocommerce_process_product_meta hook which no longer exists. It looks like it’s been replaced by a woocommerce_process_product_meta_* hook for each product type. Since we’re only using a Simple product, we’ll use the woocommerce_process_product_meta_simple hook. Now we know which hook to use, some code:


function save( $post_id ) {

    // check nonce
    if( ! ( isset( $_POST['woocommerce_meta_nonce'], $_POST['sif_ships_in'] ) || wp_verify_nonce( sanitize_key( $_POST['woocommerce_meta_nonce'] ), 'woocommerce_save_data' ) ) ) {
        return false;
    }

    update_post_meta( $post_id, '_sif_ships_in', sanitize_text_field( $_POST['sif_ships_in'] ) );

}
add_action( 'woocommerce_process_product_meta_simple', 'JB\SIF\save' );

In the save() function, (which we can name “save” without collisions because we’re using namespaces), we check the field is part of the $_POST array, check the nonce set by WooCommerce, and save the contents of the field, without forgetting to sanitize the data first.

WooCommerce may already checks the nonce for us, but I am not 100% sure, so I added an extra check.

Output the Data on the Single Product Page

WooCommerce’s plethora of hooks make placing the output relatively simple. I chose to put it right below the “Add to Cart” button on the Single Product page, but if you look through WooCommerce’s templates/content-single-product.php file and the files in the templates/single-product directory you’ll find lots hooks to use – and of course you’re not restricted to the single-product page. If you want to add your output to each product in a list of products take a look in the templates/content-product.php file.


function print_output() {

    global $product;
    $ships_in = get_post_meta( $product->get_id(), '_sif_ships_in', true );


    if( !empty( $ships_in )): ?>
        
< ?php echo sprintf( __( 'Ships in %s.', 'jb-sif' ), $ships_in ); ?>
< ?php endif; } add_action( 'woocommerce_single_product_summary', 'JB\SIF\print_output', 45 );

Our print_output() function is pretty simple: Grab the post_meta, check it’s not empty, and output it mixed in with a “Ships In” string. It’s hooked into the woocommerce_single_product_summary action with a priority of 45, which means it appear after the Add to Cart button and after the Product Meta. WooCommerce’s template files are really good at explaining what functions are hooked to actions, and the priority they’re hooked at, which makes it easy to place things on a page without having to copy templates over to your theme.

A screenshot showing the output of the "Ships In" field on the single-product page.
Here you can see the output of the new “Ships In” field on the single-product page. The product meta was moved elsewhere.

Conclusion

Adding fields to the WooCommerce part of a WordPress admin area is much easier than adding a whole metabox. You can grab the completed plugin on Github. You’ll see there’s a bit more to the plugin: requirements are checked using Mark Jaquith’s method before embarking on a namespaced plugin with shorthand array syntax.

If this is useful let me know, and feel free to open tickets and send pull requests on Github.

A PHP Version Switcher for the AMP stack on OS X

A friend is thinking about buying MAMP Pro so he can test on different versions of PHP. Since I just set up version-switching myself, for free, here’s how I did it. Hopefully it’ll save someone a bit of money or frustration.

This is happening on a Mac, with the latest version of OS X, (Currently 10.11.3 El Capitan), using the pre-installed version of Apache, and Homebrew as a package manager.

Step 1: Install more than one version of PHP on your system

Using Homebrew you can install a version of PHP, then “unlink” it so it’s not currently being used:

brew install php7
brew unlink php7

If you want to be able to switch PHP versions on the command-line there’s a tool called PHP Version that does the heavy lifting for you, and with Homebrew it’s easy to install.

Step 2: Make it easy to change the PHP version Apache is using

First, stop the main Apache config, (/etc/apache2/httpd.conf), file from loading PHP. Find any line that mentions PHP modules and comment it out, (put a # at the beginning of the line). For example:

#LoadModule php5_module libexec/apache2/libphp5.so
#LoadModule php5_module /usr/local/opt/php56/libexec/apache2/libphp5.so

Now we need an Apache config file for each PHP version that you want to run. I’ve put these files in /etc/apache2/other/ and used the file extension “.conffile” to prevent Apache from auto-loading them:

A screenshot of /etc/apache2/conf/other

Each of your .conffile files needs to have the location of the PHP library, and any extra PHP info. This is the contents of my php-7.conffile:

LoadModule php7_module /usr/local/opt/php70/libexec/apache2/libphp7.so
<IfModule php7_module>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
</IfModule>
<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

php-5.conffile is very similar, but has the path of the PHP 5 .so file and the IfModule conditional checks for php5_module.

If you have any existing php*.conf files in /etc/apache2/other/ they can be blank.

Next we need to make an easy way to tell Apache to use one of the .conffiles, which I’ve  done by creating a symbolic link to the one I want to use. Since I don’t like typing out ln commands I’ve created a script, which I call php-v.sh.

#!/bin/bash
if [ $1 = 7 ]; then
PHP_VERSION=7
else
PHP_VERSION=5
fi
`sudo ln -Fs /etc/apache2/other/php-$PHP_VERSION.conffile /etc/apache2/other/php.conf`
`sudo apachectl graceful`
echo "PHP version switched to $PHP_VERSION"

This script symlinks /etc/apache2/other/php.conf to the .conffile that I want to use, (using the -F switch to overwrite the old symlink), and restarts Apache. It needs sudo so it’ll ask for your password.

Finally, it’s nice to have php-v.sh available from everywhere, so I’ve created a symlink from ~/bin/php-v, which is in my PATH, to the actual script:


ln -s /Users/John/Developer/Scripts/php-v.sh /Users/John/bin/php-v

And now I can switch PHP versions at will:A screenshot of my terminal, having just switched PHP versions quickly & easily!

Apache/cPanel Log Analysis using MySQL

Every once in a while I need to analyze log files from a web server. Sometimes people ask my what is sucking up all the bandwidth on their hosting plan, other times bots are driving them crazy so we try to identify easy ones to block. Sometimes we want to know how many times a certain URL, or group of URLs was loaded. The logs I work with usually come from LAMP servers running cPanel on a shared host but the techniques below are adaptable to any format by adjusting the import query.

Tools:

I usually use Sequel Pro to run queries. Even when writing the queries myself I prefer a GUI to the command line. It is much easier to inspect tables visually and my query history is saved. That said, we’re working with plain SQL here, (maybe with a bit of MySQL flavour), so any MySQL client will work.

You also need a MySQL server to process the queries. I use the same one as I use for development on my laptop, if you use MAMP, WAMP, or any other AMP you should have access to a local MySQL server.

Part 1: Setup

Before analyzing the data it must be imported, and before importing data there needs to be a table to put it in. Here is the CREATE TABLE statement I use:

 CREATE TABLE `analysis_job` (
`ip` varchar(16) DEFAULT NULL,
`clientid` varchar(128) DEFAULT NULL,
`userid` varchar(128) DEFAULT NULL,
`timestamp` varchar(128) DEFAULT NULL,
`offset` varchar(128) DEFAULT NULL,
`request` varchar(250) DEFAULT NULL,
`status` int(128) DEFAULT '0',
`size` int(128) DEFAULT '0',
`referrer` varchar(250) DEFAULT NULL,
`useragent` varchar(250) DEFAULT NULL,
`ts` datetime DEFAULT NULL,
`uts` bigint(10) DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

A few of these columns may need more explanation:

  • clientid: something Apache creates, almost alway blank
  • userid: something else Apache creates almost always blank
  • request: the contents of the HTTP request, includes the HTTP method, (usually GET or POST), the URL, and the HTTP version
  • status: the HTTP Status code the server sent
  • size: the size of the response. I believe this is in bytes.
  • ts: A timestamp with better formatting than the timestamp column to make it easier to do date operations, (we’ll compute this after importing the data)
  • uts: A Unix Timestamp representation of the ts column, handy for sorting.

Each line of the log file is a single HTTP request. The import statement tells the MySQL server that the fields are enclosed by double quotes and delimited by spaces, from left to right. This means that the timestamp gets broken into two columns, hence the timestamp and offset columns in the CREATE TABLE statement above.

And here’s a sample log entry for a request from Bingbot, (this is a single line from the log file, it is wrapped to multiple lines here):

127.0.0.1 - - [30/Nov/2014:04:00:45 -0800] "GET /directory/page.html HTTP/1.1" 200 9499 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

For this request the column values are:

ip:
127.0.0.1
clientid:
userid:
timestamp:
[30/Nov/2014:04:00:45
offset:
-0800]
request:
GET /directory/page.html HTTP/1.1
status:
200
size:
9499
referrer:
useragent:
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The WHM/cPanel based servers that I have encountered always use this log format, but if the server you are working with is different you will have to modify the CREATE TABLE statement to match it.

Part 2: Importing Data

With the table set up it’s fairly quick & easy to import the data:

#Import data
LOAD DATA LOCAL INFILE '/path/to/logfile/logfile' 
INTO TABLE analysis_job
FIELDS TERMINATED BY ' ' ENCLOSED BY '"' LINES TERMINATED BY "\n";

This loads everything into the table. Now we have to extract an actual date from the timestamp column and turn it into a datetime for the ts column using MySQL’s STR_TO_DATE() function:

# to make the string into an actual datetime
UPDATE analysis_job SET ts = STR_TO_DATE(timestamp, '[%d/%b/%Y:%H:%i:%s');

Now there’s a small problem to solve: If the system timezone setting on the server is different from the timezone setting in MySQL on whatever computer you’re using as a database server, and if you want to be able to ask “what happened today in my timezone?” we need to shift the content of the ts column to match your timezone. You can see the UTC offset of the server by looking at the offset column. In my case the original server is on Pacific Time, and I’m on Eastern Time, so we need to add 3 hours to ts:

# adjust timezone
UPDATE analysis_job SET ts = DATE_ADD(ts, INTERVAL 3 HOUR);

If you want to have the Unix Timestamp version of ts available to you then you need to set it:

# Set unix timestamp column
UPDATE analysis_job SET uts = UNIX_TIMESTAMP( ts );

Part 3: Analysis

Here’s the fun part. If you’re good with the ORDER BY and GROUP BY clauses of SQL, and their corresponding functions, you can find all sorts of interesting stuff. Here are some examples that I’ve used:

Find the total bandwidth used during the time your log file spans:

SELECT SUM(size) as totalsize, (SUM(size) / 1048576) as MB FROM analysis_job;

Or for the last week, (change the WHERE clause for different timeframes):

SELECT SUM(size) as totalsize, (SUM(size) / 1048576) as MB FROM analysis_job WHERE ts > DATE_SUB( NOW(), INTERVAL 1 WEEK );

Find the requests, (and therefore files), that use the most bandwidth:

SELECT request, SUM(size) as totalsize, (SUM(size) / 1048576) as MB FROM analysis_job GROUP BY request ORDER BY totalsize Desc;

Do you suspect there’s one computer out there hammering your site? Find the IP that’s using the most bandwidth:

SELECT ip, SUM(size) as totalsize, (SUM(size) / 1048576) as MB FROM analysis_job GROUP BY ip ORDER BY totalsize Desc;

Or the User-Agent:

SELECT useragent, SUM(size) as totalsize, (SUM(size) / 1048576) as MB FROM analysis_job GROUP BY useragent ORDER BY totalsize Desc;

Who is hammering you with the most requests?

SELECT ip, COUNT(*) as requests FROM analysis_job GROUP BY ip ORDER BY requests Desc;

Again, group by User-Agent to find the program, (assuming the User-Agent isn’t spoofed):

SELECT user agent, COUNT(*) as requests FROM analysis_job GROUP BY user agent ORDER BY requests Desc;

Any of these can be restricted by date, like we did with bandwidth above.

You can also zoom in on one User-Agent, for example:

SELECT * FROM analysis_job WHERE useragent='Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)';

Or certain file types:

SELECT * FROM analysis_job WHERE request LIKE '%.css%'; # CSS
SELECT * FROM analysis_job WHERE request LIKE '%.html%'; # HTML

Mix it up for detailed analysis. This finds the IPs that have sent the most hits in the past 5 hours that don’t declare themselves as bots and are accessing normal parts of the website, (not cPanel, stats packages, robots.txt, etc):

# Non-bot IPs that have sent the most hits in the last 5 hours
SELECT COUNT(*) AS cnt, ip FROM analysis_job 
WHERE
ts > DATE_SUB(NOW(), INTERVAL 5 HOUR) 
AND useragent NOT LIKE '%bot%' 
AND useragent NOT LIKE '%spider%' 
AND useragent NOT LIKE 'Mediapartners%' 
AND request NOT LIKE '%/api/1/%' 
AND request NOT LIKE '%/media/%'
AND request NOT LIKE '%/pub/%'
AND request NOT LIKE '%cpanel/%'
AND request NOT LIKE '%mint/%'
AND request NOT LIKE '%/robots.txt%'
GROUP BY ip
ORDER BY cnt Desc;

Part 4: Other Considerations

While this data can help find interesting traffic patterns and trends it is important to remember that IP addresses can be spoofed or proxied and User-Agents can be set to whatever a client wants so those values cannot be entirely trusted.

In addition, if you use CloudFlare, the IP address will almost always be the IP of a CloudFlare server unless your host has mod_cloudflare, or some equivalent, installed.

Even with these caveats server logs are an untapped information mine, and having a chunk of time in a database table makes it much easier to dig in that mine.