Jörn Berkefeld:
i dont think we can remove those after all. if we use explicit definitions they either get removed or (in case of SOAP api) requested.
both outcomes would be undesirable.

--> concluding that for now we need to keep the deletes in there.

other thoughts, Douglas Midgley?

https://help.salesforce.com/articleView?id=mc_as_automation_studio_activities_reference.htm&type=5

section:
Advanced Configuration

what:
define all attributes
define what PatternType is inside of pattern

side quest:

• check if fields like typeId and timezoneId can be removed or at least only re-added on deploy via catalyst
• check if icalRecur can be generated automatically based on pattern and endDate (icalRecur is required on deploy, tested already!)

Create basic Electron UI to display how a UI could look.
Functionally should retrieve and document. Deploy is future story

can be tested on ACN Learning > catalyst target 1

First of all, thanks for this great work, it will be some time before I get to hopefully use it, but this is really a great step forward in SFMC development.

I was just curious to read more in the developer documentation you mention at the end of the readme, but the documents the links seem to be referring to are missing (there is also no docs folder in the repo). Are these docs not ready yet or are they accidentally missing and can they be added?

thinking of the same output as for CSCLSROZ-229:

columns:

• metadata type
• location = file path & name (starting after "retrieve/" in the filepath)
• external key
• name
• Action: add, update, delete, error
• Error Message

at the same time, remove those custom per-type logs currently generated

details: https://developer.salesforce.com/docs/atlas.en-us.noversion.mc-apis.meta/mc-apis/getting-started-spec.htm

list of types: https://help.salesforce.com/articleView?id=mc_jb_activity_type_reference.htm&type=5

Subtasks:

• Remove filter for jb-created automations from automation.definition / CSCLSROZ-535 #1288
• #1287
  • Add JB Decision Split / CSCLSROZ-465
  • Add JB Einstein Frequency Split / CSCLSROZ-468
  • Add JB Einstein Scoring Split / CSCLSROZ-467
  • Add JB Engagement Split / CSCLSROZ-464
  • Add JB Random Split / CSCLSROZ-466
• Add JB Content Builder In-App Activity / CSCLSROZ-474
• Add JB Content Builder LINE Carousel Activity / CSCLSROZ-472
• Add JB Content Builder LINE Multi-Content Activity / CSCLSROZ-477
• Add JB Content Builder Push Notification Activity / CSCLSROZ-479
• Add JB Content Builder SMS Activity / CSCLSROZ-481
• Add JB Email Activity / CSCLSROZ-457 #62
• Add JB Google Analytics SMS Metrics / CSCLSROZ-483
• Add JB In-App Push Activity / CSCLSROZ-473
• Add JB Inbox Push Activity / CSCLSROZ-475
• Add JB Join Activity / CSCLSROZ-469
• Add JB LINE Carousel Activity / CSCLSROZ-471
• Add JB LINE Multi-Content Activity / CSCLSROZ-476
• Add JB Mobile Activity Tracking / CSCLSROZ-484
• Add JB Push Notification Activity / CSCLSROZ-478
• Add JB SMS Activity / CSCLSROZ-480
• Add JB SMS Analytics / CSCLSROZ-482
• Add JB Wait-Until-Date Activity / CSCLSROZ-462
• Add JB Wait-by-Attriubte Activity / CSCLSROZ-463
• Add JB Wait-by-Duration Activity / CSCLSROZ-461

MCC specific subtasks:

• Add JB Campaign Member Activity / CSCLSROZ-486
• Add JB Case Activity / CSCLSROZ-491
• Add JB Contact Activity / CSCLSROZ-490
• Add JB Update-a-Contact Activity / CSCLSROZ-470
• Add JB Account Activity / CSCLSROZ-489
• Add JB Convert Lead Activity / CSCLSROZ-487
• Add JB Lead Activity / CSCLSROZ-492
• Add JB Opportunity Activity / CSCLSROZ-493
• Add JB Task Activity / CSCLSROZ-488
• Add JB Object Activity / CSCLSROZ-485

Create a report to list all permissions of each user to allow for a comprehensive view

Describe the bug
imagine SFMC's api server is not available or your are throttled or your personal internet connection keeps dropping out. Currently that can cause a seemingly endless download loop.

Expected behavior
only the subtype that caused the issue is restart, or better yet: only the particular item inside of the subtype is retried.

Screenshots

Versions:

• mcdev (mcdev --version): 3.0.1

It seems Marketing Cloud's API servers have become... less usable. I've seen "ECONNRESET" errors across the board and independent of the metadata type during retrieve operations. We need to build in more handlers for that situation deeper in the code. Right now, it's mostly handled on a type-by-type basis which does not help much when you try to download hundreds of entries and it always fails at the very end, causing a complete restart.

Versions:

• mcdev (mcdev --version): 3.0.1

Describe the bug
Retrieving automations fails and stops all imports after automation.

To Reproduce
Steps to reproduce the behavior:

1. CLI: mcdev retrieve X/Y automation
2. Output:
   13:54:53 info: Retrieving: automation
   13:55:09 error: Connection error. Retrying automation
   13:55:25 error: Connection error. Retrying automation
   13:55:42 error: Connection error. Retrying automation
   13:55:57 error: Connection error. Retrying automation
   13:56:12 error: Connection error. Retrying automation
   ...

Logs:
13:54:53.880 info: Retrieving: automation
13:55:09.585 error: Connection error. Retrying automation
13:55:09.586 debug: Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
13:55:25.514 error: Connection error. Retrying automation
13:55:25.516 debug: Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
13:55:42.779 error: Connection error. Retrying automation
13:55:42.779 debug: Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)
13:55:57.985 error: Connection error. Retrying automation
13:55:57.985 debug: Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:209:20)

Expected behavior
Retrieving automation works.

Screenshots
na

Versions:

• mcdev (mcdev --version): 3.0.1
• npm (npm --version): 6.14.13
• node (node --version): v14.17.1
• Operating system (Windows/Mac): Mac

Describe the bug
if you have only one asset subtype in the config, that's what will be retrieved. if you have none in it, none will be retrieved

Expected behavior
selectively retrieving asset falls back to retrieving the defaults

Versions:

• mcdev (mcdev --version): 3.0.1

idea is to give add warning codes into the "missing folder" errors for some types and link to the docs for a detailed description

• dataExtension --> table still exists, data still exists, but not visible in UI.
  > how to fix: re-deploy the dataextensions with base path OR use catalyst delete
• automation --> still exists but only visible via "all automations" instead of in subfolders
  > automation activities (e.g. query) --> like automation
  > how to fix: select resources in UI and move to existing folder OR move via change of folder path in jsons and re-deploy

Subtasks:

SOAP

• Delete DataExtension
• Delete DataExtensionField
• Delete EmailSendDefinition
• Delete List
• Delete TriggeredSend

REST

• Delete Asset
• Delete DataExtract
• Delete FileTransfer
• Delete ImportFile
• Delete Query
• Delete Script

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

the current approach does not actually use the files committed to git but retrieves again from the server -that might break things given that DEV might have already changed by the time this runs.

therefore, buildTemplate must be added which does what retrieveAsTemplate does, only completely local, without a retrieve

change .catalystrc to .mcdevtoolsrc

Requirement

1. Add new command "restore [name]"
2. Command should work on one type at a time and on one BU
3. Command should list metadata that is hidden with a selectable list, showing name and folder id, like the one we use in select types
4. The user then opts in to every entry that shall be reset to main folder
5. System then creates / clears the deploy folder for that particular BU
6. System copies the selected meta data into the right deploy subfolders
7. System asks if it may deploy now (yes/no question like the one used on 2nd execution of "init")
8. System deploys (or not)

The default folder for each type should be set in our definition.jsons. If none was set there, the "restore" command will assume the type is not supported. This can then be used to output a list to the user to pick from when running the command without parameters.

Side-note: The same definitions-based "pick-type"/"supported-type" approach can then be established for "delete", "document", "retrieveAsTemplate" and "buildDefinition", making it easier to use them all. --> the method for type selection could for instance take an attribute on the type-definition to understand if it supports a type (supports: { "retrieve": true, "restore": false, "delete":true, ... }

Background
We currently see 3 types of errors pop up on most projects with "lost" parent folders being one of these. This has proven to be irritating to users and how to deal with errors happens to be one of the most asked questions i have received so far.

https://developer.salesforce.com/docs/atlas.en-us.noversion.mc-apis.meta/mc-apis/filterdefinition.htm
https://developer.salesforce.com/docs/atlas.en-us.noversion.mc-apis.meta/mc-apis/filteractivity.htm 

should cover

• journey
• filter activity
• filtered dataExtensions? see #63

situation:
Case A: Dev/Prod

• we have 2 BUs: "Dev-BU" and "Prod-BU"
  • ACN development is solely done on "Dev-BU"
  • potential difficulty: the client may make changes directly on "Prod-BU"
  • testing is conducted on "Dev-BU"
• branches (Uwe Völlger - please review) (ignore upper half)

  

  • feature/ — changes to "Dev-BU", no automated deployment
  • master — upon commit, deploy diff to "Prod-BU"
• branch rules (like in the above picture)
  • no direct commits to release + master branch --> only via PR
  • feature -> master
  • feature branches created off of master branch

Comments from Uwe:
That is fine
End comments from Uwe

Case B: Dev/Qa/Prod

• we have 3 BUs: "Dev-BU", "Qa-BU" and "Prod-BU"
  • ACN development is solely done on "Dev-BU"
  • potential difficulty: the client may make changes directly on "Prod-BU"
  • testing is conducted on "Qa-BU"
• branches (Uwe Völlger - please review) (ignore lower half)

  

  • feature/ — changes to "Dev-BU", no automated deployment
  • release/ — upon commit, deploy diff to "Qa-BU"
  • master — upon commit, deploy diff to "Prod-BU"
• branch rules (like in the above picture)
  • no direct commits to release + master branch --> only via PR
  • feature -> release -> master
  • feature branches created off of release branches
  • release branches created off of master branch

Comments from Uwe:
Here I would do it differentyl. The normal development happens in the "lower half". Release branches are created as late as possible - that means only if there are parallel releases. This typically happens when bugs must be fixed found during a QA phase while development happens already on the next release. This helps to create as less branches as possible, as each branch also means merges, that might result in conflicts and a lot of work.
So branch rules would be

Uwe Völlger: falls mehr envs nötig werden würde ich zu branches übergehen die nach den envs benannt sind.
Comments from Uwe:
I would like to avoid env-specific branches. It would work, but would create much effort. What's about the idea which we have in SF classic: we define a release (basically a range of commits) and deploy the same to QA BU and more BUs. That of course means the BUs must basically the same, and differences must somehow covered.
End comments from Uwe

Der klassische develop branch könnte bei langfristigeren projekten aber genauso zur anwendung kommen? ziel outcome sollte sein, dass die branching strategie möglichst simpel ist, damit auch nicht versierte nutzer schnell die logik dahinter verstehen UND schätzen

Comments from Uwe:
+**+Ja, das würde auch für weniger agile Projekte passen. Im Prinzip ist das dann auch nicht viel anders als ohne den Branch. Details im Umgang stehen in https://kxdocuments.accenture.com/Contribution/130a8ab6-37c0-43b9-9a9f-a18b16f03e79
End comments from Uwe

Background
Creating tables often takes more time than necessary due to an insufficient interface. Also, when you are not online, you cannot work on it. Also, double-checking if certain tables exist on a BU takes minutes just for switching to that BU and loading Email-Studio / Contact Builder.
On top, some features work only in one interface, while others only in the other (e.g. update External Key only works in Email Studio, similar for adding a new field with isNullable = false)

Requirement

• Based on a previous retrieve we show the folder-dataextension structure for all BUs + shared on an instance.
• Users can:

1. Search for DE names, DE external keys, field names
2. Create new DEs in a selected BU & folder
3. Update existing DEs
4. Delete new (not uploaded) DEs again.
5. Revert to previous versions

Roadmap prioritization
Essentially this user story rebuilds the SFMC interface – however, the main gain here is the offline capability.
Also, being able to search the entire instance for information within 1 second should be considered a transparency boost.

Note
We might want to split this into multiple user stories as the effort is high.
Highly depends on first UI to be published.
Can be worked on by multiple team members after initial search-interface was created.

Support

- https://developer.salesforce.com/docs/atlas.en-us.mc-apis.meta/mc-apis/senderprofile.htm

- https://developer.salesforce.com/docs/atlas.en-us.mc-apis.meta/mc-apis/sendclassification.htm

- https://developer.salesforce.com/docs/atlas.en-us.mc-apis.meta/mc-apis/deliveryprofile.htm

first test if this is still happening in 2.4.0 dev-version

from docs -> var inquirer = require('inquirer');
inquirer
.prompt([
/* Pass your questions in here */
])
.then(answers => {
// Use user feedback for... whatever!!
})
.catch(error => {
if(error.isTtyError) {
// Prompt couldn't be rendered in the current environment
} else {
// Something else when wrong
}
});

so the following does the same thing.
const responses = await inquirer.prompt(questions);

QueryAllAccounts and shared data extensions sometimes cause issues with retrieve for caching and general retrieve.

relevant for standard contact import via mc connect and alike

https://help.salesforce.com/articleView?id=mc_as_automation_studio_activities_reference.htm&type=5

code = 0 --> all ok

code = 1 --> error happened

how to: https://stackoverflow.com/questions/5266152/how-to-exit-in-node-js

devops solutions can listen for that end then show the build status correctly. right now, the build status is always green/OK because we handle all errors gracefully and by node apps default seem to send code 0

https://help.salesforce.com/articleView?id=mc_as_automation_studio_activities_reference.htm&type=5

instead of detailed logs we could show a progress bar to users.
meanwhile, we still log everything to our log file in logs/ for detailed analysis.
and with --verbose or --debug flag we show more info directly in console instead

other idea: https://github.com/sudheerj/awesome-javascript-technologies/blob/master/01-Inquirerjs/examples/10_bottom-bar.js

if you run catalyst upgrade on a project-folder after cloning the repo it will fail on upgrading .catalystrc (the config file) but continue doing other things.

either silently ignore this part or error out completely.

--> could be a general error if the auth file is missing. should fail and ask to run catalyst init

expected:

deploy folder and retrieve folder should show the same json for a deployed file, except for created/lastsaved info.

actual: 

left = deploy; right = retrieve folder (after deploy was executed)

the next retrieve gets the normal format back

we already got a working solution when it just comes to simple conversion. references to other contentAreas are NOT yet auto-converted.
capturing the internal development here in this new issue.

new command is convertClassic:

the current approach does not actually use the files committed to git but retrieves again from the server -that might break things given that DEV might have already changed by the time this runs.

therefore, buildTemplate must be added which does what retrieveAsTemplate does, only completely local, without a retrieve

add fixKeys command that re-uses the --changeKeyField logic together with this.definition.nameField

this is a basic wrapper around --changeKeyField that searches for metadata of the given type with name/key mismatches.

Acceptance Criteria:

• types that cannot be used with this method are identified by:
  • use id as key (this.definition.keyField === this.definition.idField)
  • use the same field for name and key (this.definition.keyField === this.definition.nameField)
• this command should search offline --> do a full retrieve of the BU first before executing the actual find logic.
• after the key-change we need to then iterate over the definition/*.definition.js files to check for types that are dependent on the type you just fixed the keys for

out of scope:

• rewriting assets / scripts based on new key values (handled in #1001)

basic steps:

1. retrieve selected type
2. find items with key != name (metadata[(this.definition.keyField] !== metadata[(this.definition.nameField])
3. run deploy cred/bu type --fromRetrieve --changeKeyField=${this.definition.nameField} on found keys for selected type
4. iterate over elements in lib\MetadataTypeDefinitions.js to check array in x.definition.dependencies for matches with the selected type
   • do notice that there are sometimes subtypes defined here as well, eg. "asset-message" --> if you applied fixKeys to "asset" then this would be a match
   • run full retrieve on types that have our selected type as a dependency to ensure that we have keys properly updated in our local folder

... the user then has to git add / commit & push on their own.

it seems the way the "last checked" time is compared it currently fails on "first check".
lets make sure we also run this on whatever first run we execute to ensure old but never used installs produce immediate update notifications

likely requires an update of the dependency itself rather than one inside of catalyst

with npm 7 it looks like this:

still worked with npm  6.x and before:

happens when running mcdev init

unfortunatley this not only happens for our config files but also for ALL files initially downloaded from BUs... means one gets a LONG list of stupid errors

with git version 2.28.0 on windows:

idea ist to use our deploy folder to create a package that can then be deployed via SF's deployment manager UI.

makes it easier to deploy components from our library to new clients without the need for CLI operations

Extend/Update Scripts for Testing for future integration to Jenkins.

Test if Business Units can be created/deleted by API

https://help.salesforce.com/articleView?id=mc_as_automation_studio_activities_reference.htm&type=5

possible solution:
https://salesforce.stackexchange.com/a/386492/28940

Describe the bug
ECONNRESET during retrieving campaign not handled:

11:58:26 info: Retrieving: campaign
(node:27896) UnhandledPromiseRejectionWarning: Error: Error: read ECONNRESET
at C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\metadataTypes\Campaign.js:25:27
at Campaign. (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\node_modules\sfmc-fuelsdk-node\lib\objects\Campaign.js:53:4)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:93:5)

To Reproduce
Steps to reproduce the behavior:

1. start retrieving campaigns
2. cut your network connection in between
3. See error

Expected behavior
It should keep retrying

Screenshots

Versions:

• mcdev (mcdev --version): 3.1.0
• npm (npm --version): 7.8.0
• node (node --version): 14.16.0
• Operating system (Windows/Mac): win10

we should add support for audit trails & convert the result into a nice html or md file to allow actually making it readable

we can already extract a csv apparently using data extracts/automations though.

https://help.salesforce.com/articleView?id=mc_overview_audit_trail.htm&type=5
https://developer.salesforce.com/docs/atlas.en-us.noversion.mc-apis.meta/mc-apis/getAuditEvents.htm
https://help.salesforce.com/articleView?id=mc_overview_audit_trail.htm&type=5

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

When running createDeltaPackage it sometimes causes to crash when it retrieves DataExtensions with fuel-soap.

Log:

info: Retrieving as Template: dataExtension
events.js:292
throw er; // Unhandled 'error' event
^

Error: Error: Invalid argument for the equals operator. Filter[0] cannot be null or empty.
at /node_modules/mcdev/lib/metadataTypes/MetadataType.js:532:35
at FuelSoap. (/node_modules/fuel-soap/lib/fuel-soap.js:131:6)
at FuelSoap. (/node_modules/fuel-soap/lib/fuel-soap.js:603:5)
at Parser. (/node_modules/xml2js/lib/parser.js:304:18)
at Parser.emit (events.js:315:20)
. . . . . . .

Describe the bug
during retrieve of dataExtension the dataExtensionField caching can cause total failure.

11:54:53.241 info: Caching dependent Metadata: dataExtensionTemplate
11:54:53.474 info: Retrieving: dataExtension
11:55:12.684 debug: SOAP.retrieve Error: read ECONNRESET
11:55:12.685 info: - Caching dependent Metadata: dataExtensionField
11:55:13.451 error: Cannot read property 'Fields' of undefined
11:55:13.451 debug: TypeError: Cannot read property 'Fields' of undefined
at C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\metadataTypes\DataExtension.js:306:59
at Array.forEach ()
at Function.retrieve (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\metadataTypes\DataExtension.js:304:23)
at async C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\Retriever.js:102:34
at async Object.retryOnError (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\util\util.js:682:13)
at async Retriever.retrieve (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\Retriever.js:101:21)
at async _retrieveBU (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\index.js:683:13)
at async retrieve (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\index.js:616:17)
11:55:13.452 error: Retriever.retrieve:: Retrieving dataExtension failed
11:55:13.452 debug: Error: Cannot read property 'Fields' of undefined
at Object.retryOnError (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\util\util.js:717:23)
at async Retriever.retrieve (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\Retriever.js:101:21)
at async _retrieveBU (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\index.js:683:13)
at async retrieve (C:\Users\joern.berkefeld\AppData\Roaming\npm\node_modules\mcdev\lib\index.js:616:17)

To Reproduce
Steps to reproduce the behavior:
have network issues during dataExtension retrieve

Expected behavior
auto retries

Screenshots

Versions:

• mcdev (mcdev --version): 3.0.1

right now it's still using mcdev\node_modules\sfmc-fuelsdk-node\lib\objects\Campaign.js
with this.client.campaign and this.client.campaignAsset

Describe the bug
it does not actually save any files

To Reproduce

Expected behavior
Files are stored successfully

Versions:

• mcdev (mcdev --version): 3.0.1
• npm (npm --version):
• node (node --version):
• Operating system (Windows/Mac):

Conversion of Data Extension Retention DataRetentionPeriodUnitOfMeasure to use mapping in definition file. Currently this uses a integer which is not human understandable