Only 6.0 or less is now supported

Hi!
With CDH 6.0.1 i have the following issue:
Following the Guideline I tar the Parcel with gzip. Then I provide the parcel via my repo Server and paste the URL into CM Parcel Configurations -> Remote Parcel Repository URLs
Download Fails because of the hash verification.

The downloaded parcel is stored in /opt/cloudera/parcel-repo The .parcel is gunzipped (and I think therefore the hash verification Fails)

If I do not gzip the .parcel, then the Step target/validator.jar -f Fails. But Cloudera Manager accepts the .parcel and distributes it.

So what is the solution?

I am wondering if there an end-end example showing parcel, CSD with metrics integration?

I have a Spring boot based application that I want to deploy as a CSD. I can export the metrics to JMX or StatsD but trying to figure out how it would connect with the MDL to the Cloudera Dashboard.

Hi!

I tried this with CDH 6.0.1 and even cannot build the validator. mvn install runs into an unknown error.

Anyone tried it with CDH 6 ?

Thx!

I tried to use the "alternatives" key in my parcels.json. However, Cloudera's parcel validiator kicked it out with:

==> Unrecognized field "scripts.alternatives". Recognized fields are "[defines, _defines

My CSD file is valid according to the local test. But the CM-logs show:

2015-06-13 11:21:01,884 INFO main:com.cloudera.cmf.cdhclient.CdhExecutorFactory: Adding options to bad options list:
2015-06-13 11:21:09,249 WARN main:com.cloudera.csd.components.CsdLocalRepository: Skipping [/opt/cloudera/csd/TTFAQ-1.1.0.jar]: { The service version [1.0] must match [1.1.0] for file [TTFAQ-1.1.0] }

I had this issue once, but if you do not do CSD related you might miss this detail.

Maybe we should not only validate the service.dsl file, but also the final JAR?

I created the below service.sdl, however log_dir doesn't get created. It is a parameter of type "path"

{
  "name": "STORM",
  "label": "STORM",
  "description": "storm",
  "version": "0.9.4.5",
  "runAs": {
    "user": "storm",
    "group": "storm"
  },
  "icon": "images/icon.png",
  "parcel": {
    "requiredTags": [
      "storm"
    ]
  },
  "serviceDependencies": [
    {
      "name": "ZOOKEEPER",
      "required": "true"
    }
  ],
  "parameters": [
    {
      "name": "topology_max_spout_pending",
      "label": "topology.max.spout.pending",
      "description": "topology.max.spout.pending",
      "type": "long",
      "default": 10,
      "min": 1,
      "max": 50
    },
    {
      "name": "log_dir",
      "label": "log dir",
      "description": "log dir",
      "type": "path",
      "default": "/var/log/storm",
      "pathType": "localDataDir",
      "required": true,
      "mode": "0755"
    }
  ],
  "roles": [
    {
      "name": "STORM_SUPERVISOR",
      "label": "storm supervisor",
      "pluralLabel": "storm supervisors",
      "jvmBased": true,
      "startRunner": {
        "program": "scripts/control.sh",
        "args": [
          "supervisor"
        ],
        "environmentVariables": {
          "LOG_DIR": "${log_dir}"
        }
      }
    },
    {
      "name": "STORM_UI",
      "label": "storm ui",
      "pluralLabel": "storm ui",
      "jvmBased": true,
      "startRunner": {
        "program": "scripts/control.sh",
        "args": [
          "ui"
        ],
        "environmentVariables": {
          "UI_PORT": "${ui_port}",
          "LOG_DIR": "${log_dir}"
        }
      },
      "externalLink": {
        "name": "storm_ui",
        "label": "storm Web ui",
        "url": "http://${host}:${ui_port}"
      },
      "topology": {
        "minInstances": "1",
        "maxInstances": "1"
      },
      "parameters": [
        {
          "name": "ui_port",
          "label": "ui.port",
          "description": "ui.port",
          "type": "port",
          "default": 9999
        }
      ]
    },
    {
      "name": "STORM_NIMBUS",
      "label": "storm nimbus",
      "pluralLabel": "storm nimbus",
      "jvmBased": true,
      "startRunner": {
        "program": "scripts/control.sh",
        "args": [
          "nimbus"
        ],
        "environmentVariables": {
          "LOG_DIR": "${log_dir}"
        }
      },
      "topology": {
        "minInstances": "1",
        "maxInstances": "1"
      },
      "parameters": []
    }
  ]
}

After adding a dependency on the SPARK_ON_YARN service type to our custom CSD, it no longer passes the validator. However it works fine in CM (confirmed on 5.5, 5.7). It seems the validator is missing this valid service type

Validating: src/descriptor/service.sdl
==> SPARK_ON_YARN must be a valid service type

We need from user to enter/edit long config value. Currently we use "type" : "string" but in UI it's very inconvenient, need to move cursor. Would be great to have a text type that would rendered as textarea in ui.

In the SDL page under Parameters -> Type -> string_enum it lists valid_values as a valid key. It should be validValues.

https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference

I'm developing a custom CSD. Although I have added "gateway" to service.sdl, I do not see a "Deploy Client Configuration" command in the "Action" menu. Could you please help?

Is there a way to select the same host for all roles in services with one click in the UI while starting the service instead of selecting the host for each role one by one?
Say there are 4 roles in the new service and the cluster has 3 hosts, can I just select some option like "select host1 for all roles?"

According this link: https://blog.cloudera.com/blog/2013/07/how-does-cloudera-manager-work/
It said:

To do its monitoring, Cloudera Manager collects metrics. Metrics are simply numeric values, associated with a name (e.g., “cpu seconds”), an entity they apply to (“host17”), and a timestamp. Most of this metric collection is done by the agent. The agent talks to a supervised process, grabs the metrics, and forwards them to the service monitor. In most cases, this is done once per minute.

But I can't find document about report custom metric to supervised or any other service.
How to report custom metric in CSD?

Do not use the following reserved names for the name of a file:
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these names followed immediately by an extension; for example, NUL.txt is not recommended. For more information, see Namespaces.

is there any introduction for configuring alternatives.json, and maybe better for all meta-config?
what will be happened after configured the property "destination" and "source"?
what's the difference between configuring property "isDirectory" true and false?

I was getting the below error all the time.
git clone https://github.com/prateek/nifi-parcel.git
Initialized empty Git repository in /etc/nifi-parcel/.git/
error: while accessing https://github.com/prateek/nifi-parcel.git/info/refs

fatal: HTTP request failed

can you explain pre requisite step as well clearly?
Thanks.

have created a csd for GRAFANA, with logging type:glog.
the log is not visible on CM 5.6 for the new role.

I am writing a CSD that supports multiple roles and each role does a spark submit job on startup. Is a way to stop these yarn spark jobs when I stop the service.
To describe the problem more accurately, is there a way for me to track the roles under the service and the yarn spark jobs under each role from the command line so that I can write a custom stoprunner for the service that will stop the yarn jobs associated with that role?

Thanks in advance.

I now want to integrate ES services, Parcel support?

https://github.com/cloudera/cm_ext/wiki/The%20alternatives.json%20file

The alternatives.json page describes the natural behaviour of the update-alternatives command is the destination is already occupied. However it does not talk about a situation in which it is currently occupied by an existing alternative by the same name, but the destination differs.

An example is the famous API differences in Guava between 11 and 16.

A Parcel may be employed to upgrade an application in which the Guava dependency changes. Thus naturally, the alternatives may change from.

    "guava" : {
      "destination": "/opt/application/lib/guava-11.0.2.jar",
      "source": "lib/guava-11.0.2.jar",
      "priority": 10,
      "isDirectory": false
    },

To

    "guava" : {
      "destination": "/opt/application/lib/guava-16.0.1.jar",
      "source": "lib/guava-16.0.1.jar",
      "priority": 10,
      "isDirectory": false
    },

Making use of automatic Parcel deactivation when activating a Parcel of the same name will fail.

Debug from the Cloudera Agent reports an error from update-alternatives.

the primary link for guava must be /opt/application/lib/guava-11.0.2.jar

Accordingly, the doc should highlight the additional importance that destination must be equal between Parcel versions (if the parcel is responsible for the alternative entry)

i.e. destination is version independent of the entry and should be static between Parcel versions.

    "guava" : {
      "destination": "/opt/application/lib/APP_guava.jar",
      "source": "lib/guava-16.0.1.jar",
      "priority": 10,
      "isDirectory": false
    },

Hi, guys!
Please tell me about "roles" tag and inner tags in service.sdl. Do you have some plans for describing this tag in wiki or maybe there is some other docmentation.

Hello,

How do we stop individual roles without stopping the whole service?
I tried using stopRunner similar to the startRunner, however noting seems to be working.

The loggingType currently only supports log4j and logback. Programs using log4j2 have a different .properties syntax, and seem to prefer the .xml format.

The validator is really useful for finding mistakes in the CSD and cutting down on the test cycle. It's small and fast and a good candidate to run as part of the build. However, it's not easy to integrate with the build.

To make it run in a portable way as part of the JAR build process, I had to check it out using maven-scm-plugin, build it using Maven and exec-maven-plugin, and then manually run it on my manually specified SDL files using exec-maven-plugin. The relevant Maven pom.xml file parts are attached at the bottom.

It would be much simpler to do if the validator had a public Maven repository. I could get rid of the first two steps, and more easily cache the results using my local Maven repository.

It would be even simpler if there was a Maven plugin in a public Maven repository that could automatically find any SDL files and validate them as part of the test phase.

  <properties>
    <skip.build.validator>false</skip.build.validator>
  </properties>

  <profiles>
    <profile>
      <id>validator-ready</id>
      <activation>
        <file>
          <exists>target/checkout/validator/target/validator.jar</exists>
        </file>
      </activation>
      <properties>
        <skip.build.validator>true</skip.build.validator>
      </properties>
    </profile>
  </profiles>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-scm-plugin</artifactId>
        <version>1.9.4</version>
        <executions>
          <execution>
            <phase>test</phase>
            <goals>
              <goal>checkout</goal>
            </goals>
            <configuration>
              <connectionUrl>scm:git:https://github.com/cloudera/cm_ext.git</connectionUrl>
              <url>https://github.com/cloudera/cm_ext</url>
              <skipCheckoutIfExists>true</skipCheckoutIfExists>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>1.4.0</version>
        <executions>
          <execution>
            <id>build-validator</id>
            <phase>test</phase>
            <goals>
              <goal>exec</goal>
            </goals>
            <configuration>
              <skip>${skip.build.validator}</skip>
              <executable>mvn</executable>
              <workingDirectory>target/checkout</workingDirectory>
              <arguments>
                <argument>-Dmaven.test.skip=true</argument>
                <argument>package</argument>
              </arguments>
            </configuration>
          </execution>
          <execution>
            <id>run-validator</id>
            <phase>test</phase>
            <goals>
              <goal>exec</goal>
            </goals>
            <configuration>
              <executable>java</executable>
              <arguments>
                <argument>-jar</argument>
                <argument>target/checkout/validator/target/validator.jar</argument>
                <argument>-s</argument>
                <argument>src/descriptor/service.sdl</argument>
              </arguments>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

I created a parcel to distribute some files on a cluster. I had no need for environment variables so I omitted the env script. After validating and activating my parcel, CM would fail to start any CDH roles. The STDERR output showed that the PARCEL_DIRNAME was using the name of my parcel instead of CDH. This made it impossible for agent scripts to locate bigtop.

It seems that in /usr/lib/cmf/service/common/cloudera-config.sh source_parcel_environment assumes that $SCRIPT_ARRAY and $DIRNAME_ARRAY are congruent. Because my parcel.json omitted scripts.defines, these arrays were incongruent and so CDH_DIRNAME got set to my parcel's name in cdh_env.sh.

The analysis in the preceding paragraph could be wrong. The upshot is that everything worked again after I added a defines script to my parcel. The parcel validator should not have approved a parcel without the required script.

This looks like it's related to #4. When I run mvn install in the validator directory from a fresh checkout, I see this error:
[ERROR] Failed to execute goal on project schema-validator: Could not resolve dependencies for project com.cloudera.enterprise:schema-validator:jar:5.4.0: Could not find artifact com.cloudera.cmf.schema:cloudera-manager-schema:jar:5.4.0 in cloudera-external (https://repository.cloudera.com/artifactory/ext-release-local/) -> [Help 1]

I can see schema-validator-5.4.0.jar in libs-staging-local, but nothing in ext-release-local.

Hi,

I successfully built the cm_csds from the root with maven. I also built cm_ext from the root but am getting this error:

[foo@bar target]$ $JAVA_HOME/bin/java -jar $VALIDATOR -s KAFKA-5.8.0.jar 
Validating: KAFKA-5.8.0.jar
==> Unexpected character ('P' (code 80)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: [B@76e992d; line: 1, column: 2]

This occurs for at least ECHO and KAFKA.

I'm using Cloudera's java,

[foo@bar target]$ echo $JAVA_HOME
/usr/java/jdk1.7.0_67-cloudera

using my hand-built validator,

[foo@bar target]$ echo $VALIDATOR
$HOME/cloudera/cm_ext/validator/target/validator.jar

and using a fresh yum install of maven

[foo@bar target]$ mvn -v
Apache Maven 3.0.5 (Red Hat 3.0.5-16)
Maven home: /usr/share/maven
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_67-cloudera/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.36.3.el7.x86_64", arch: "amd64", family: "unix"

I searched but didn't find any issues, and I'm fairly green using maven, so I apologize if this is something obvious/general.

Thanks!

According to https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference#-commands-1, requiredRoleState is required.

But empirically, leaving it out appears to work fine, allowing the command to be run when the role is either stopped or running. Is this just a typo in the docs? Would like to confirm. Tested on CM 5.8.

Hello,
I was trying to build the validator using maven 3, but it seems that something is missing I checked out also https://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException to be sure, butthe problem was not there:

The main error I have is :
Could not find artifact com.cloudera.cmf.schema:cloudera-manager-schema:jar:5.0.0 in cloudera-external.

I ran mvn install in /validator first as I thought it was enough, then I ran it with the -X option, but I had still the error.

So I understood that I had to run mvn install in /cm-schema

Then I ran again mvn install in /validator and it worked :)

I think you should put a little line troubleshooting for people like me who encountered this tiny problem :p

Thanks a lot for your work btw, have a great day :D

The current version, the validator for xenial shows warning.
https://github.com/cloudera/cm_ext/blob/master/validator/src/main/java/com/cloudera/cli/validator/components/ParcelFileRunner.java#L66-L79