This will ensure the code is more testable

Perhaps a new option

--remediate

could be set that would allow certain checks to auto remediate themselves. E.g. if you expected a certain module to be enabled, and it is not, then it could enable the module for you.

Would be nice to be able to supply a file to which should be contained in the Drupal variable robotstxt.

Example of the default value from Drupal 7 core:

#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html

User-agent: *
Crawl-delay: 10
# CSS, JS, Images
Allow: /misc/*.css$
Allow: /misc/*.css?
Allow: /misc/*.js$
Allow: /misc/*.js?
Allow: /misc/*.gif
Allow: /misc/*.jpg
Allow: /misc/*.jpeg
Allow: /misc/*.png
Allow: /modules/*.css$
Allow: /modules/*.css?
Allow: /modules/*.js$
Allow: /modules/*.js?
Allow: /modules/*.gif
Allow: /modules/*.jpg
Allow: /modules/*.jpeg
Allow: /modules/*.png
Allow: /profiles/*.css$
Allow: /profiles/*.css?
Allow: /profiles/*.js$
Allow: /profiles/*.js?
Allow: /profiles/*.gif
Allow: /profiles/*.jpg
Allow: /profiles/*.jpeg
Allow: /profiles/*.png
Allow: /themes/*.css$
Allow: /themes/*.css?
Allow: /themes/*.js$
Allow: /themes/*.js?
Allow: /themes/*.gif
Allow: /themes/*.jpg
Allow: /themes/*.jpeg
Allow: /themes/*.png
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/

Get the following error when running either an individual site audit or ACSF audit:
PHP Fatal error: Class 'SiteAudit\Ssh\Command' not found in /home/paulk/scripts/site-audit/src/Executor/ExecutorRemote.php on line 20

This occurs whenever we try to run an SSH command. e.g. all of the ACSF calls, or the Theme size check for an individual site.

I've tried on the following setups:

• Ubuntu 12.04 LTS, php 5.6
• Ubuntu 16.04 LTS, (php 7 & php5.6)

Could this be composer/autoloader related?

We should have a check to ensure that the timezone is correctly set, I would think to a specified whitelist of timezones.

$ drush --uri=https://www.sitename.gov.au/ vget date_default_timezone
date_default_timezone: Australia/Canberra

Twig is at least 342 times better than PHP template, we should use it.

This leads to a lot of 404s

Would be nice to breakdown the requests by type, e.g. HTML, images, JS, CSS etc

e.g. node, taxonomy_term. Would also be great to measure the largest database tables, say the top 3 or something.

All govCMS SaaS sites are required to have:

• metatag enabled
• metatag configured to have a canonical URL which points at production

variable govcms_ckan_endpoint_url

It would be great to allow a --report=report.html optional option to the command to write the output to an HTML file, for easy PDF generation.

It is time that these non-default profiles move to their own repositories (either public or private). govCMS already has one setup in https://github.com/govCMS/audit-site

This variable should be false.

Bad for performance, the message appears for admins:

For easier theme development, the theme registry is being rebuilt on every page request. It is extremely important to turn off this feature on production websites.

So hopefully we can avoid manually setting , etc in strings.

In 1.x, checks use doctrine annotations to define check metadata. It was done this way so that the metadata and the Check class itself were together in a single file. In this way it made checks more portable. But there were however, some downsides:

• In profiles, checks were specified by their Class and namespace which made them hard to compile and a DX barrier for contributors
• Annotations use a pseudo code syntax which requires the actual metadata to be passed in in datatypes like strings. This means the data needs to be escaped which makes readability harder.
• Text editors don't natively interpret annotations in comments providing no syntax highlighting
• PHP does not parse comments and cannot find syntax errors (though doctrine provides this). This i more of an issue in Drupal than in Drutiny.
• Multi-line data in annotations is ugly because of the * at the begining of each line. It makes management of these difficult.
• Annotations look better for defining small pieces of data and the plan in 2.x is to use metadata more efficiently.
• Adding HTML and markdown inside of annotations looks dirty and hard to read. It just feels like the wrong place to put annotations.

In 2.x, I re-introduced YAML as the metadata file for Checks. While this does decouple check metadata from the checks, it does make adoption from a DX standpoint easier (IMO)

• YAML is a more broadly used and common format. It doesn't require a predefined Annotation class to use it (this is also the reason why YAML is validated in runtime).
• The filename itself introduces a new naming format that is more DX friendly.
• YAML is more likely to be supported in text editors
• YAML has great multi-line support and doesn't hinder HTML or markdown and neither will require escaping.

One of the key differences between the two approaches is how they address structured data.

• In Annotations, you define a class and only properties of that class can be provided in the annotation.
• In YAML, there are no rules but the rules of the syntax, there is no guarantee of data structure.

Because check metadata requires structure, I also added to 2.x YAML validators at runtime which essentially is doing the same thing as Doctrine would be doing for Annotations. So this not to replace TravisCI validation but to provide more responsive feedback to Check developers and to allow more assumptions to be made in the framework about the data provided.

@fiasco has started a v2 branch master...fiasco:2.x

Key new features:

• Checks have defined options now, complete with validation and default values. This helps to make this explicit, so things like profile builders could ask questions in the future.
• Driver plugins - the ability to execute the check via a plugin system
• Sandbox class, instead of Context
• New console commands, check:run etc
• Progress bar on CLI

Still needs discussion:

• Removing the annotation from the check class, I was quite a fan of this as it kept the checks self contained. Drutiny in the early days actually had separate YAML files, and it was hard to keep the YAML and the check together, and made PRs harder to review. Essentially the crux of this issue is having multi line annotation strings.
• Validation of the check at runtime vs TravisCI time, I personally prefer TravisCI to do this checking.

Still needs to be done:

• Caching of drush commands (e.g. variable-get)
• Porting of all the checks over
• Porting of the profiles
• Moving the namespace of checks out of a D7 and D8 folder structure and into the types of checks they are, e.g. 'security', 'performance', 'bestpractice'
• port the phpunit tests over
• Removal of settings based checks, and only support the check classes (as they contain additional metadata)
• Drupal multisite and ACSF running

.htaccess should do something with this URL, to ensure that Drupal is not forced to bootstrap in order to serve the 404. It is highly unlikely that .xml is in the fast 404 extension list, so quite often this 1 path will be thousands of bootstraps/day that you don't need.

/autodiscover/autodiscover.xml

Potential rewrite rules:

RewriteRule ^autodiscover/autodiscover\.xml$ - [L,NC,R=404]

When administrators are found, they should allow remediation to remove them.

This should override the profile's defined checks.

Ideally we should keep the domain the user is browsing on, rather than force redirecting to the production domain.

$ drush vset securepages_basepath ''
$ drush vset securepages_basepath_ssl ''

And not have things like slashes in them.

Spotted in one of the checks

$ drush @SITENAME sqlq "SELECT fc.field_name, fc.data, fci.data FROM field_config fc JOIN field_config_instance fci ON fc.id = fci.field_id WHERE fc.type = 'entityreference'"
Warning: Using a password on the command line interface can be insecure.

The output string Warning: Using a password on the command line interface can be insecure. really messes up the output. Please cleanse.

Also use it in the HTML report

I think now the amount of information we need to convey now exceeds a single page. I would love to integrate with github pages or similar using the docs folder https://github.com/blog/2233-publish-your-project-documentation-with-github-pages.

Another option is gitbooks.io

I don't see anything in the provided documentation about how to install drutiny. There's information on installing some of its requirements, but not on how to install drutiny itself.

Hi.
Your tool is included in the site audit for govCMS. While running those checks on a site that had no special GA configuration, I received the D7 GA error. It would be much more helpful if it told you what its actually expecting you to have in the "Code snippet (after)" field. Pull request coming shortly to do just that.
Thanks for your work.

Warning: array_flip(): Can only flip STRING and INTEGER values! in site-audit/src/Profile/Profile.php on line 122
PHP Warning:  array_flip(): Can only flip STRING and INTEGER values! in site-audit/src/Profile/Profile.php on line 122

I think the structure of the command array may have changed. array_flip expects a single dimension array so it can swap keys with values. We get the above warning because the command array is a multi-dimensional array.

Example Command array item.

["SiteAudit\Check\Phantomas\PageWeight"]=>
  object(__PHP_Incomplete_Class)#38 (11) {
    ["__PHP_Incomplete_Class_Name"]=>
    string(30) "SiteAudit\Annotation\CheckInfo"
    ["title"]=>
    string(11) "Page weight"
    ["description"]=>
    string(188) "You should aim to keep your page weight as low as possible to ensure speedy download and rendering times for your site. This impacts not only your site's user experience but also it's SEO."
    ["remediation"]=>
    string(47) "Look to optimise the largest and slowest files."
    ["success"]=>
    string(111) "Page weight is smaller than <code>:max_size</code> MB. Actual size is <code>:value</code> MB. :biggest_response"
    ["failure"]=>
    string(120) "Page weight is currently larger than <code>:max_size</code> MB. Actual size is <code>:value</code> MB. :biggest_response"
    ["exception"]=>
    string(32) "Could not determine page weight."
    ["not_available"]=>
    NULL
    ["warning"]=>
    string(157) "Page weight is smaller than <code>:max_size</code> MB but larger than <code>:warning_size</code> MB. Actual size is <code>:value</code> MB. :biggest_response"
    ["notice"]=>
    NULL
    ["supports_remediation"]=>
    bool(false)
  }

Ideally we will ensure that some of the newer extensions are covered, extensions to look at covering by default:

• gif
• jpe?g
• png
• svg
• ttf
• rtf
• cfm
• css
• js
• ics
• docx?
• xlsx?
• pptx?
• nsf

Extensions you should never whitelist

This can be due to wanting to setup the redirect module for legacy paths.

• xml
• php
• html?
• aspx?
• ashx

Stock Drupal 7

$conf['404_fast_paths_exclude'] = '/\/(?:styles)|(?:system\/files)\//';
$conf['404_fast_paths'] = '/\.(?:txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

Proposed setup

The default parameters should be similar to:

Images:

• png
• gif
• jpeg OR jpg
• svg
• tiff
• bmp
• raw
• webp

Office:

• doc OR docx
• xls OR xlsx
• ppt OR pptx

Misc binary files:

• swf
• flv
• cgi
• dll
• exe
• nsf
• cfm
• ttf

Misc ASCII files:

• bat
• pl
• asp
• ics
• rtf

$conf['404_fast_paths_exclude'] = '/\/(?:styles)|(?:system\/files)\//';
$conf['404_fast_paths'] = '/\.(?:png|gif|jpe?g|svg|tiff|bmp|raw|webp|docx?|xlsx?|pptx?|swf|flv|cgi|dll|exe|nsf|cfm|ttf|bat|pl|asp|ics|rtf)$/i';

Phantomas can do this, there is a jsErrors key in the Offenders section

The output for the entity reference check shows the machine name for the field- but checks configurations per field definition and doesn't tells us exactly which field on which content type the configuration is broken- we should be able to update particular field configurations with a query to ease this.

Field conf looks like

'label' => 'Topics',
'widget' => array(
  'weight' => '6',
  - 'type' => 'options_select',
  - 'module' => 'options',
  + 'type' => 'entityreference_autocomplete',
  + 'module' => 'entityreference',
  'active' => 1,
  'settings' => array(
    'apply_chosen' => '1',
    'flatten_options' => 0,
    'flatten_sort' => 1,
  +  'match_operator' => 'CONTAINS',
  +  'size' => 60,
  +  'path' => '',
  ),
),

It might be a good idea now that the project has some CI that we could add a build that would create a drutiny.phar file that could be downloaded from the Github releases page.

Can use this https://github.com/box-project/box2 (which can be included via composer) and a composer script that will build the project into a phar.

I have done this in the past with

find . -name "*.module" -type f | grep -oe "[^/]*\.module" | cut -d"." -f1 | sort | uniq -c | sort -nr | awk '{if ($1 > 1) print "[" $1 "] - " $2}'

Should be easy to port this over to a check.

Would be nice to host an example HTML report somewhere for people to see as well. A picture is worth 1000 words.

This is Drupal's internal page cache which is often not needed when you use varnish with Drupal 7

e.g.

• video files (.mp4, .flv)
• malicious files (e.g. .bmp, .scr, .exe)

Make it configurable.

It should exit early for all subsequent ACSF theme checks.

At the moment phantomas assumes HTTPS but it would be nice to be able to make this configurable in the profile.

Howdy, neat tool! Fantastic to have more techniques and approaches to a common problem.

However, it's named the same thing as the existing utility https://www.drupal.org/project/site_audit - https://github.com/fluxsauce/site_audit

I respectfully request that you name it something distinct, as currently it's literally named the same thing as my existing project.

Traditional site audits (e.g. the checklist API module or the site_audit module) in Drupal rely on having certain modules or code present on the server in order to gather the required metrics. The main issue is that if you fail to even have these modules enabled at all, then no auditing will take place in the first instance.

site_audit doesn't depend on any third-party modules, it integrates optionally with them if they exist, and it's not a module, it's a drush command with some initial Drupal Console support as well.

Thanks!

Especially around video files. Also potentially other extensions

tar.gz
.tgz
.zip

Potentially related https://www.drupal.org/node/2853493 - from https://www.drupal.org/project/module_missing_message_fixer

Also the d.o docs https://www.drupal.org/node/2487215