Split composer install options into an array and ad an option to configure them. Like the rSync task $options array.

At least output a hint that they should be added manually

There is an inconsistency concerning the CreatePackageStatesTask:

The PHPDoc block says that This task is meant to be used for local packaging only...

/**
 * Creates the package states file and removes all not active packages from the according folders
 * This task is meant to be used for local packaging only
 */
class CreatePackageStatesTask extends AbstractCliTask
{
    ...
}

...but TYPO3\Surf\Application\TYPO3\CMS::register() registers this task to be executed after the transfer tasks.

Is the PHPDoc block wrong or is the task registered incorrectly?

I don't have PHP 7 installed local and I don't want to fake composer platform with
"config": { "platform": { "php": "7.0" } }
so I use docker
e.g. docker run --name composer -v $(pwd):/app t3easy/composer:alpine install

If i set this as composerCommandPath the $(pwd) will be escaped as it should be.

What about using some $replacePaths for composerCommandPath like in shell task especially for $manifestPath to do something like
$composerCommandPath = 'docker run --name composer -v {manifestPath}:/app t3easy/composer:alpine'

I tried to add and remove tasks following the documentation, but it didn't work.
In the docs it is suggested to do
$workflow = new \TYPO3\Surf\Domain\Model\SimpleWorkflow();
to access the SimpleWorkflow object to add new tasks. However this obviously creates a complete new workflow, it does not allow you to modify the add tasks to the exiting one, as it seems to be implied in the docs.
To modify the existing workflow, it seems that you need to access the existing workflow with:
$workflow = $deployment->getWorkflow();
and it may be needed to be execute it in the onInitialize handler like:

$deployment->onInitialize(function () use ($deployment, $application) {
    // set file permission
    $workflow = $deployment->getWorkflow();
    // ...
}

If this is correct it would be helpful to update the documentation in this sense.

Can db5c9d1 please be reverted? It causes some problems:

1. The BaseApplication::addSymlink() method still uses the other order and creates the symlink array with wrong array keys / values.
2. It might occur that you have multiple symlinks to the same file. This can not be provided in the array because the source path can only occur once:

$array = (
  'index.php' => 'typo3_src/index.php'
  'otherpath/index.php' => 'typo3_src/index.php'
);

TYPO3 CMS uses the option initialDeployment directly from the deployment
https://github.com/TYPO3/Surf/blob/master/src/Application/TYPO3/CMS.php#L69
It's better to use the option from the combined inherited option array.
This way it's not possible to add applications to the stack that use the same option:

$deployment->setOption('initialDeployment', false);
/** var \TYPO3\Surf\Application\TYPO3\CMS $typo3 */
$typo3->setOption('initialDeployment', true);
/** var \What\Ever\Application $app */
$app->setOption('initialDeployment', false); // or inherit from deployment

While activating an TYPO3 CMS extension the static sql data is imported. Some extension rely on reimporting this data with up to date contents. Probably the best known example is "static_info_tables" and its language specific brothers and sisters. "static_info_tables" comes with an update script, which rewrites the table contents.

IMO it is necessary that those updates can be done automatically while a deployment.

Basically there are two options:

1) New feature for TYPO3 core and extension configuration

If an extensions is activated for the first time, the static sql data is imported into the database and a value is set in the table "sys_registry", that the import happened. If it is set to "1", the table is not touched any more.

An option for the core could be, that the extension author defines whether the static data should be updated or not, when updating the extension.

2) Surf & typo3_console

A more simpler and faster to achieve solution might be to make it configurable for developers / integrators using surf and the typo3console.

The extensions that must / will have their static sql data reimported, could be defined in an array or a comma separated list in the options of surf. The import itself is done via typo3_console and the TYPO3 core InstallUtility and its functions "processDatabaseUpdates" or "importStaticSqlFile".

My favorite would be no. 2 as it would enable the feature for existing TYPO3 versions. But no. 1 would be the cleaner way.

What do you think?

Ciao
Marcus

Current tasks running 'typo3/cli_dispatch.phpsh' will fail because they get executed in the project root directory instead of the webDirectory (if that is in use, of course).

i.e. https://github.com/TYPO3/Surf/blob/master/src/Task/TYPO3/CMS/FlushCachesTask.php#L59 will fail if caches should be cleared using coreapi.

Every task should have a documentation block as class annotation

We have a staging system that is only accessible with htaccess protection.
Executing the WebOpcacheResetExecuteTask fails obviously since it uses file_get_contents.
Would it make sense to use Curl here? See http://stackoverflow.com/a/24045706

Create a new workflow for initializing a deployment on remote nodes. This workflow should:

• Create deployment path (try mkdir -p) if it does not exist yet
• Create all needed directories (or at least clear cache if existing)
• Ask to create node specific environment files
• Optionally move data (assets, db) to nodes

Not sure if I'm really right here! (sorry)

Im just trying to get typo3/surf working with my neos site for many hours now. When I fire "composer require typo3/surf:dev-master" on the console at my neos instances root directory the packages content gets installed to Packages/Libraries/typo3/surf

Because of this no ./flow surf:* commands are available on the cli. What's the matter here? Could please anybody either provide some help or point me to the correct place where I may get help?

Thank you guys very much!

(Below is a screenshot where the typo3/surf files are located after composer require command)

symfony/console v3.0.1 requires php >=5.5.9

Hey there,

we've moved some submodules with
git mv oldSubmoduleHome/MySubmodule newSubmoduleHome/MySubmodule
for the first deployment everything works very well, but the second deployment crashes after the git module stuff from
Surf/src/Task/Git/AbstractCheckoutTask.php:93 without a specific error. (Got exception "Command returned non-zero return code: 1" rolling back)

We investigated the problem and here is our finding, hope it helps.

After the git mv of a submodule the .gitmodules is modified by git in the following way,

[submodule "oldSubmoduleHome/MySubmodule"]
    path = oldSubmoduleHome/MySubmodule
    url = https://github.com/ME/myApp.git

[submodule "oldSubmoduleHome/MySubmodule"]
    path = newSubmoduleHome/MySubmodule
    url = https://github.com/ME/myApp.git

As you can see, just the path is changed to the new location, everything else stays untouched. Especially the submodule name in the header.

Back to the problem in SURF:
We think one of the tasks in Surf/src/Task/Git/AbstractCheckoutTask.php:93 uses the name instead of the path and therefore the whole stuff breaks.

#WORKAROUND
In .gitmodules change the line
[submodule "oldSubmoduleHome/MySubmodule"]
to
[submodule "newSubmoduleHome/MySubmodule"]
and push the changes. The next deployment should work.

Locally you have to do the following:
$: git submodule deinit .
$: git submodule init
$: git submodule update

After that everything should work as expected.

Thanks in advance

git version 2.7.4
TYPO3 Surf version 2.0.0-beta7

The problem is in
https://github.com/TYPO3/Surf/blob/master/src/Task/Git/AbstractCheckoutTask.php#L93

&& for mod in `git submodule status | awk '{ print $2 }'`; do git config -f .git/config submodule.\${mod}.url `git config -f .gitmodules --get submodule.\${mod}.url` && echo synced \$mod; done

The first deploy works because the affected line will be called on subsequent deploys only (when the checkout is updated).

The code seems to expect the name from the submodule with
git submodule status

For me the output is the path of the submodule. For example:

e55fba0e95b4e6cbc0d915b6683cb8464188f77 packages/myext (heads/master)

Therefore packages/myext is extracted as ${mod} which means
git config -f .git/config submodule.${mod}.url
will fail, because we need the submodule name here (eg. packages-myext)

I can get the submodule name with
git config --list | grep '^submodule\.'
which output
[email protected]:example/myext.git

I never used surf with older git versions and submodules but the guess is that the behaviour changed which breaks the code.

Occassionaly it is possible that while a depoyment is running and the current symlink got deleted, that the webserver would create the directories in order to write to typo3temp. Thus a current/.../typo3temp structure would have been created, preventing surf to create the current symlink pointing to the new release.

Why does this script write the opcodecache file to localhost?
I think it has to be placed on remote so that I can execute it with the WebOpcacheResetExecuteTask.

If deployment scripts are found in Build/Surf move them to ~/.surf/deployments/

Scripts inside the scripts > pre-deploy array should run automatically.
https://github.com/helhum/TYPO3-Distribution/blob/master/composer.json#L49
This could be used to trigger e.g. npm to build an application inside a clean environment.
The post-install-cmd that is triggered by composer automatically, could hold other tasks.
A use case:

"scripts": {
	"post-install-cmd": [
		"# composer install composer.lock file present",
		"cd web/typo3conf/ext/sitepackage && npm install && gulp build"
	],
	"post-update-cmd": [
		"# composer update or composer install composer.lock NOT file present",
		"cd web/typo3conf/ext/sitepackage && npm install && gulp build"
	],
	"pre-deploy": [
		"cd web/typo3conf/ext/sitepackage && npm prune --production"
	]
},

The additional npm prune --production command deletes all devDependencies (necessary for the build) which are not necessary on production and should not be transferred to the remote node but keeps the normal production dependencies.
https://docs.npmjs.com/cli/prune

As of now, Surf does not support asking for the sudo password during deployment. However, sometimes it is necessary during deployment (depending on your server infrastrukture / security considerations etc.) to execute a sudo command.

It would really be great to have this feature, not least because Surf's "bigger brother" Capistrano also supports it.

When using the password option with surf.phar 2.0.0-beta4 I run into this error

couldn't read file "/Private/Scripts/PasswordSshLogin.expect": no such file or directory

The deployment should start php scripts (like cli_dispatch.phpsh) by using the interpreter, not as executable.

/usr/bin/env php ${site} extbase varnishinstance:update ${iplist}

Reason: Some secured systems are using mount options "nodev,noexec,ro" for the typo3src folder.

https://github.com/TYPO3/Surf/blob/9eef93038f5300c5d39c02a205c8015dd0271e9f/README.md#shell-task-option-expansion

Add

• deploymentPath
• releasePath
• sharedPath
• currentPath
• previousPath

Most notably:

• Use command to migrate deployment definitions
• Move deployment definitions to ~/.surf/
• Default deployment strategy changed from remote git to rsync

I had to change the git history.
After this, I tried to deploy but the command
https://github.com/TYPO3/Surf/blob/master/src/Task/Git/AbstractCheckoutTask.php#L61
returns two sha1 checksums and therefore the preg_match https://github.com/TYPO3/Surf/blob/master/src/Task/Git/AbstractCheckoutTask.php#L62
fails and an execution is thrown.

To continuously deploy small changes there should be a option to keep releases by age.

given you have an deployment step like:

$workflow->defineTask(
    'sbs.lapo:fixHtaccess',
    'TYPO3\\Surf\\Task\\ShellTask',
    array(
        'command' => 'sed -i "s/RewriteBase \/.*/RewriteBase \/staging\/ \n SetEnv FLOW_CONTEXT Production /g" {releasePath}/Web/.htaccess'
    )
);

Then the escapeshellarg in https://github.com/TYPO3/Surf/blob/2.0.0-beta6/src/Task/ShellTask.php#L41 will break the command as the path is not concatenateable.

Even worse the replacement in the rollbakc step returns different data, as there is no escape ...
https://github.com/TYPO3/Surf/blob/2.0.0-beta6/src/Task/ShellTask.php#L87

The default setting for Flow is this:

protected $options = array(
    'packageMethod' => 'git',
    'transferMethod' => 'rsync',
    'updateMethod' => 'composer'
);

This would do a git checkout, then transfer the data to the server and then run composer install.
This means also that other post install tasks would run on the target node.

I would propose to change this back to the normal Application default like this:

protected $options = array(
    'packageMethod' => 'git',
    'transferMethod' => 'rsync',
    'updateMethod' => null
);

Because this would do a local git checkout, then run composer and then transfer everything to the target node and only create the release.

A problem is also that for example setting the packageMethod to git and leaving the updateMethod to composer would run composer install two times as the Flow app automatically adds the composer install task depending on those methods.

Looking into this also reveals for me that it's a bit difficult to understand from those options what actually happens...

Right now it is set to --exclude .git and can not be changed. It would make sense to add .gitignore and .gitmodules or, even better, make it configurable. .git could stay as a default.

Make the private key that should be used for SSH connections (both shell and rsync) configurable.

(moved from https://forge.typo3.org/issues/66505)

Read class annotations from tasks and create documentation from that

Hi,

I've created an SCP Task for the flow-based version of Surf.
https://gist.github.com/DavidSporer/7485da2e83421cb06292

Does it make sense to include this to the core of this new version or is it something that most people don't need?

Regards,
David

I want to use the smoke test. Therefore I have to configure my Apache to use the /releases/next path.
Unfortunatly after a deployment the next-Path is deleted.
Therefore reloading apache will fail.
Is there a best practice for that?

This is strange because the random script name should be alphanumeric according to the code.
Solved it temporarily by setting the scriptIdentifier.

stage-xyz (my-app) TYPO3\Surf\Task\Php\WebOpcacheResetCreateScriptTask
PHP Warning:  file_put_contents(/home/xyz/.surf/workspace/stage-xyz/my-app/Web/surf-opcache-reset-du/gpfSNxj+cVc95vTbz1oJ/HGw/Y4x7.php): failed to open stream: No such file or directory in phar:///home/xyz/ci-builds/a17d0ae5/0/vendor/my-app/surf.phar/src/Task/Php/WebOpcacheResetCreateScriptTask.php on line 71
Got exception "Could not write file "/home/xyz/.surf/workspace/stage-xyz/my-app/Web/surf-opcache-reset-du/gpfSNxj+cVc95vTbz1oJ/HGw/Y4x7.php"" rolling back.

The TYPO3\Surf\Task\TYPO3\CMS\CopyConfigurationTask switches the extension from .yml to .php but doesn't change the target path.

https://docs.typo3.org/surf/TYPO3SurfDocumentation/ from https://typo3.org/typo3-cms/additional-products/surf/

I have a 2nd File Storage downloads
and placed the folder under shared/Data/downloads like shared/Data/fileadmin.
I set the option directories for the SymlinkDataTask
$application->setOption('TYPO3\\Surf\\Task\\TYPO3\\CMS\\SymlinkDataTask[directories]', ['downloads']);

I would expect the symlink inside the web folder but it will be created in the root of the project.

composer.json
composer.lock
downloads -> ../../shared/Data/downloads
vendor
web

As a workaround I moved the data to shared/Data/web/downloads and set option to
$application->setOption('TYPO3\\Surf\\Task\\TYPO3\\CMS\\SymlinkDataTask[directories]', ['web/downloads']);

The Task tries to create directories shared/Data/Logs, shared/Data/Persistent and shared/Configuration, but with change 642f7c3 the base directory for creating these directories changed from the deployment base to the release directory.

I could not find a specific reason for mentioned change, should we fix the Flow task or revert said change?

These lines of code don't make sence:
https://github.com/TYPO3/Surf/blob/master/src/Task/TYPO3/CMS/CreatePackageStatesTask.php#L33-L41

I have not installed typo3_console in typo3conf/ext/ but I indeed have typo3cms in my root folder which is symlinked to vendor/helhum/typo3-console/Scripts/typo3cms.

The exception should only be thrown if BOTH things are false!

Every deployment needs to set flow version to 3.0 because default is still 2.0 which is quite old.

Neos site import is not working as the wrong command is used

./flow neos.flow:site:import

instead

./flow neos.neos:site:import

should have been use.

Currently the autoload.php file is not found when installing the surf tool with composer.

I would suggest we use the same approach as phpunit:

Move the surf binary in the root folder and use a loop for searching the autoload.php file:

foreach (array(__DIR__ . '/../../autoload.php', __DIR__ . '/../vendor/autoload.php', __DIR__ . '/vendor/autoload.php') as $file) {
    if (file_exists($file)) {
        include($file);
        break;
    }
}

The encryption command still does not work and is bound to Flow.
We need to decide wether to remove it or migrate to symfony console command

Right now, there are two options concerning the folder structure of TYPO3 CMS projects.

• applicationRootDirectory
• applicationWebDirectory

There is no reference to either of these options outside the TYPO3 CMS tasks scope.

Right now, both options seem to be used inconsistently.

Example:

• SymlinkDataTask symlinks fileadmin and user_upload in the applicationRootDirectory. This seems to indicate that applicationRootDirectory is the docroot.
• AbstractCliTask->fileExists looks for files relative to applicationRootDirectory, while AbstractCliTask->directoryExists looks for folders in applicationWebDirectory
• FlushCachesTask looks for the coreapi package in {applicationRootDirectory}/{applicationWebDirectory}/typo3conf/ext/coreapi but tries to execute the cli dispatcher in {applicationRootDirectory}/typo3/cli_dispatch.sh .

A simulation of the deployment should test all tools that are used during deployment.
Maybe with the tools which and where.exe and there last return code.
This doesn't check the version or parameters but would be a first step.
The registration of that tools should be next to the command in the task.
e.g. something like
$this->addToSimulatedTools('readlink');
Next to https://github.com/TYPO3/Surf/blob/master/src/Task/CleanupReleasesTask.php#L50

A 2nd parameter for addToSimulatedTools could be a test command, if which won't fit.

• Move classes to new namespace
• Adapt cli commands to work with older Flow versions and new ones

This is sometimes needed, if you need to deploy to a shared hosting with scary php executeable pathes or similar strange stuff.

Having the folloging task we can fix that stuff before transfer (no sed) needed and save also limited SSH connections:

class ReplaceTask extends \TYPO3\Surf\Domain\Model\Task
{

    /**
     * Executes this action
     *
     * @param \TYPO3\Surf\Domain\Model\Node $node
     * @param \TYPO3\Surf\Domain\Model\Application $application
     * @param \TYPO3\Surf\Domain\Model\Deployment $deployment
     * @param array $options
     * @return void
     */
    public function execute(\TYPO3\Surf\Domain\Model\Node $node, \TYPO3\Surf\Domain\Model\Application $application, \TYPO3\Surf\Domain\Model\Deployment $deployment, array $options = array())
    {
        $replacePaths = array();
        $replacePaths['{workspacePath}'] = $deployment->getWorkspacePath($application);

        if (!isset($options['pattern'])) {
            throw new \TYPO3\Surf\Exception\InvalidConfigurationException('Missing "pattern" option for ReplaceTask', 1311168045);
        }
        if (!isset($options['replacement'])) {
            throw new \TYPO3\Surf\Exception\InvalidConfigurationException('Missing "replacement" option for ReplaceTask', 1311168045);
        }
        if (!isset($options['file'])) {
            throw new \TYPO3\Surf\Exception\InvalidConfigurationException('Missing "file" option for ReplaceTask', 1311168045);
        }

        $fileLocation =  $options['file'];
        $fileLocation = str_replace(array_keys($replacePaths), $replacePaths, $fileLocation);

        if (!file_exists($fileLocation)) {
            throw new \TYPO3\Surf\Exception\InvalidConfigurationException('No file at "' . $fileLocation . '" for ReplaceTask', 1311168045);
        }

        file_put_contents(
            $fileLocation,
            preg_replace(
                $options['pattern'],
                $options['replacement'],
                file_get_contents(
                    $fileLocation
                )
            )
        );
    }
}

The Task is a simple wrapper for preg_replace - feedback would be awesome, if you like to include it, i would do a PR.

It would be helpful to have task that dumps the TYPO3 CMS database via typo3_console.
Per default this task should not be active to avoid problems with large DBs.

In File WebOpcacheResetCreateScriptTask.php line 61
$this->shellCommandService->executeOrSimulate($commands, $localhost, $deployment);
is called.
I think it has to be
$this->shell->executeOrSimulate($commands, $localhost, $deployment);
because in ShellCommandServiceAwareTrait the variable shell is set.