Enable device adding workflow through the add-on about hassio-zigbee2mqtt HOT 41 CLOSED

See Koenkk/zigbee2mqtt#232

from hassio-zigbee2mqtt.

I think everyone agrees it would be nice. Haven’t gotten a lot of traction over in the zigbee2mqtt repo for allowing for multiple sources for devices, which I think is the best solution (feel free to comment over there).

In the mean time, we could implement an option to “copy over” a devices.json file to its place in zigbee2mqtt on add-on startup, from the share directory. It’d be experimental and hopefully temporary until zigbee2mqtt can be modified. Probably only available in the edge version?

from hassio-zigbee2mqtt.

Thanks for all the typo reports! This is fixed too now in master

from hassio-zigbee2mqtt.

Hi,

You should newer add an empty devices.js

To use a custom devices.js, download the original, only update or add the device you want to modify and use this modified devices.js.

The original file in the add-on container will be replaced by the modified copy you supply, so adding an empty or incomplete file should not work.

from hassio-zigbee2mqtt.

Maybe we should (optionally) sync the entire zigbee-shepherd-converters config. Today I tried to add a device once more, and found out that the converter files also need to be updated if the device doesn't use already known behavior. Apart from that, there is also a file to edit Home Assistant specific mapping.

Would it be an idea to copy those files to outside the docker container on default? For someone who doesn't edit their configs, this doesn't change anything while it is easy to edit otherwise. Persistence might be an issue with updates. maybe adding an option like [overwrite/ignore updates] would solve this.

from hassio-zigbee2mqtt.

Any progress on this one?

Is there a workaround in the meantime?

from hassio-zigbee2mqtt.

@arnonel If you have host access you could do some things by attaching to the docker container. i think this requires some development to work out of the box.

from hassio-zigbee2mqtt.

It seems like the device adding workflow is basically just editing your local copy of a file in zigbee-shepherd-converters (sorry if this is inaccurate, I don't have it in front of me right now). If that's the case, might be able to just copy the file to a share subdirectory (if it's not already there) on start, and if it is there, overwrite the file within zigbee-shepherd-converters with the presumably more up-to-date one in the share dir. Off the top of my head, the first problem with this approach is that if there are updates to the module, we need to avoid overwriting them, just merging local changes without totally overwriting the file.

Thoughts?

In the mean time, it seems like just modifying the file by attaching to the container is pretty much the same way you'd do it if you weren't using the add-on, so not much difference really. I guess with the exception that you're unable to easily mount (samba) the files and thus use your text editor of choice.

from hassio-zigbee2mqtt.

where would I find the zigbee-shepherd-converters file when using this addon?

from hassio-zigbee2mqtt.

@danielwelch no need to merge or other fancy stuff. Just allow converters on share as people will do a PR after happy with changes.

Updates can override stuff

from hassio-zigbee2mqtt.

Seems like the easiest thing would be to link it to share. Otherwise would need to change zigbee2mqtt to allow the converters location to be configurable similar to what we did with data directory earlier.

EDIT: I’m reading that ln in docker gets messy very quickly

from hassio-zigbee2mqtt.

Yes, you can add same folder with config/database file and link it so anyone can edit and add support devices

from hassio-zigbee2mqtt.

I think we should allow this in the edge version only.

from hassio-zigbee2mqtt.

I don’t think linking into/out of the docker container is a good idea, and any other solution adds some complexity in terms of state tracking the converters. I think we’re running into a downside of keeping this sort of thing in source code, which is otherwise working well for zigbee2mqtt, but makes this a challenge. I think the best way is to just connect to the container and modify the necessary file just as you would if you weren’t using the add-on. That way the workflow is similar, and no worries about having multiple versions of converters (if the devices.json is copied over to the share directory, what happens when the image is rebuilt with new converters that include the users updates, which have now been merged, as well as other updates?)

from hassio-zigbee2mqtt.

It is a good idea:
1, For easy add/support new devices with hassio/hassos
2, Like hass, it support third party component and does not need PR request. Some PR request take very long time, you can see this Koenkk/zigbee-herdsman-converters#21
Some one want to run custom devices they don't want to publics with hassio/hassos
3, Mount the docker image and replace file too complicate, myself try hours without success (I am not know about docker).
4, If allow zigbee-shepherd-converters in share folder, just load the custom if file exist and load default one if the folder is empty

from hassio-zigbee2mqtt.

Thanks for the comment @dzungpv.

In response to 2) it doesn’t really matter how long pull requests take to be merged, or in your custom device example, if a pull request is made at all, to you as the user, because presumably you already have the device functioning and that’s why you’re making a pull request to add it for everyone else so that it works out of the box for others.
Regarding 4) it’s not up to the add-on where zigbee2mqtt loads converters from, the library is going to load them from the zigbee2mqtt-shepherd-converters node module. What the add-on can do is overwrite that file in-place on startup, or something similar. But then you’re left with the whole updating problem I described above. Zigbee2mqtt itself (not this add-on) could change how it loads/handles the converters, but that’s an issue for that library, not here.

Regarding 3) I think we should add a step by step guide for doing this once I’ve had a chance to play around with the idea. Because of my move I’ve been without an actual home assistant instance and won’t have one until mid August after closing on a house, but I can spin up a virtual box and walk through the process, then document it, hopefully.

from hassio-zigbee2mqtt.

Why not just add a import/export there? You load the configs from the addon and edit em and load em up back? Otherwise some lil howto get to the files from inside the docker would be very nice... I have a Osram Smart+ motion controller here I wanna add to zigbee2mqtt and am too.... Windows user to make it...

from hassio-zigbee2mqtt.

I've tried to add a couple of devices (a Trust PIR sensor and the new Tradfri smart sockets) and ran into this issue. For what I could deduce, now one would indeed need to update the local installation. Am I correct to assume that requires the container to be rebuild? I could not find a share outside the container that is used to test. For me would be fine if the instance tries to find additional device settings in /share/. This allows us to test and temporarily add devices to devices.js.

It could be a little iffy with new versions of zigbee2mqtt releasing, with possible duplicate or conflicting settings but it is less cumbersome than having to rebuild containers and loading them into Hass.

from hassio-zigbee2mqtt.

Is that not exactly how HA itself works with regard to the custom components folder, adding your own tweaked files or just trying an unreleased version from the dev. Same principle applies regarding updates, onus is on user to remove the files when a proper update comes through, and no one seems to mind the risks that go with that, don't like it, don't do it works fine.

If hassio users could create new devices themselves you'd potentially have a lot more devices added with the extra part time 'devs'.

from hassio-zigbee2mqtt.

Sounds perfect to me and probably fool proof enough to have in the main version too. Would then also allow the less brave who buy something new that isn't supported to try a file supplied by a more skilled, kindly soul who doesn't have the hardware but thinks they know the fix.

If I understood right, you're no further than binning the file and restarting to be back to square one.

from hassio-zigbee2mqtt.

Yep, I think that’s all correct.

If anyone’s eager to submit a pull request, all that’s necessary would be to check if the add-one data path contains a file devices.json and, if so, overwrite the file in place within zigbee2mqtt with that one (or something similar). Could be done in the bash run.sh script.

from hassio-zigbee2mqtt.

I think everyone agrees it would be nice. Haven’t gotten a lot of traction over in the zigbee2mqtt repo for allowing for multiple sources for devices, which I think is the best solution (feel free to comment over there).

In the mean time, we could implement an option to “copy over” a devices.json file to its place in zigbee2mqtt on add-on startup, from the share directory. It’d be experimental and hopefully temporary until zigbee2mqtt can be modified. Probably only available in the edge version?

I think that would be fine as a work around. As for a solution within Zigbee2mqtt itself I understand they like to keep things more generic, considering it also already being used by Openhab users I believe.

As for the location, would /config/ be as easy as /share/ for everyone? I think the Zwave add-on also saves some files over there and it if easier configurable through the GIT add-on as well.

I'll look into it tonight but no guaranties. I am still figuring out how Docker works and how parent folders are accessed.

from hassio-zigbee2mqtt.

As for a solution within Zigbee2mqtt itself I understand they like to keep things more generic, considering it also already being used by Openhab users I believe.

I agree and am all for keeping this generic and configurable. My proposed solution over there is to add an optional configuration item that simply provides a file path to load additional device settings from.

As for the location, would /config/ be as easy as /share/ for everyone? I think the Zwave add-on also saves some files over there and it if easier configurable through the GIT add-on as well.

Currently, this add-on does not map config/ into the add-on container. This is trivial to add, but we're already using a data_path that is in the share/ directory, so I'd like for the location to be that same directory to keep things clean. In my opinion, one of the main purposes of the share directory is to make things available to add-ons and it should be used whenever it makes sense. But that's pure opinion. I don't use the git add-on, but there's no shortage of ways to edit files.

from hassio-zigbee2mqtt.

I have been tinkering with it for awhile, but I cannot figure out what path to write the file to. The result so far can be found in my fork.
When I start a bash shell in the zigbee2mqtt container, it seems to be either /node_modules/zigbee-shephard-converters/devices.js or app/node_modules/zigbee-shephard-converters/devices.js (probably a symlink to the same file, haven't checked).
Debugging is rather annoying since the logs in HA only start after the initial setup is done. Otherwise the code seems to work outside of the HASS environment.

Do you know what path it should write to?

from hassio-zigbee2mqtt.

465bc9b adds experimental support for this in the edge branch only. It would be great if people could test this out and let me know. If all seems well, I'll make a new release with this included in the non-edge version.

To try it out, set the zigbee_shepherd_devices option to true, and make sure you have a custom devices.js file in your zigbee2mqtt add-on $DATA_PATH

@ekaats thanks for the help! The problem is that it's not in the same place in the edge version vs. the non-edge version (different directory names, see the corresponding Dockerfiles).

from hassio-zigbee2mqtt.

I've given it a try but it doesn't seem to work. I noticed that you are looking for the file devices.json rather than devices.js but adding both to the data dir doesn't change behavior. Starting bash in the docker container shows the original file in /app/node_modules/zigbee-shepherd-converters/devices.js
I can copy the file manually but haven't found a way to restart node.js without restarting the whole container.

from hassio-zigbee2mqtt.

Haha, well getting the filetype right would be a good start. I’ll need to fix that.

from hassio-zigbee2mqtt.

Hi Daniel @danielwelch - great job continuing to update your add-on. I'm trying to figure out if this is now implemented, I've had a look through a bunch of issues but not sure. Hoping one day I'll be able to use Hass.io instead of moving back to Hassbian or similar to debug new devices.

Thanks mate!

from hassio-zigbee2mqtt.

Hey @highground88, thanks.
I’d say support is experimental at this point. I’ve implemented a way to use a custom devices.js file, but haven’t tested that it works, as I don’t use this feature and haven’t had the time. So, testing and feedback is very welcome, as are pull requests. What I haven’t thought about as much is toZigbee and fromZigbee stuff. I’ve tried to discuss this over in the zigbee2mqtt repo but haven’t gotten much traction.

from hassio-zigbee2mqtt.

Thanks @danielwelch , very much appreciate your assistance even though there's no personal benefit to you. Unfortunately I am somewhat linux-challenged shall we say - so I'm probably not the guy we need to find all the bugs, although I'm hopeful that there's others out there with the necessary skills reading this that might be able to give you the support you need. I'm just hoping that for my personal Zigbee issues (still getting the Nue branded switches into HA), other talented people will get this all implemented, as it would be a shame to lose all the great Hass.io features if going back to a vanilla version is the only option.

from hassio-zigbee2mqtt.

@highground88 this doesn't stop you using hassio, it just stops you using the zigbee2mqtt add-on. Install zigbee2mqtt on another device, pi, pc, whatever, and everything else stays the same. When it's updated, just turn the stand alone one off, plug the sniffer back into your hassio pi and update the add on, job done.

from hassio-zigbee2mqtt.

Hi @danielwelch , thanks for the great work...

There is another typo which prevents this from working: run.sh tries to copy file to: node_modules/zigbee-shephard-converters/devices.js , it should be zigbee-shepherd-converters instead.

from hassio-zigbee2mqtt.

@highground88 I've used the method in the zigbee2mqtt wiki for adding/fixing devices successfully, you can give it a try - the steps are pretty straightforward once you ssh to the hass.io host.

https://github.com/Koenkk/zigbee2mqtt/wiki/How-to-support-new-devices-on-Hass.io

from hassio-zigbee2mqtt.

@ekinmeroglu Thanks for the heads up, just fixed the typo. Glad to see someone has typed up a guide, maybe I should link to that in the README

from hassio-zigbee2mqtt.

Hi @danielwelch , I've updated to 1.8 and tried the new option, the devices.js in the config path was copied and used successfully \o/

There is a small glitch in the run.sh : $ZIGBEE_SHEPHERD_DEVICES_PATH is not populated, so if devices.js was not found, the error message would not be correct.

Other than that, this option makes my life very easy.. Thanks!

from hassio-zigbee2mqtt.

Thanks for great work once again...

I've added a brief documentation on this option to the zigbee2mqtt wiki:
https://github.com/Koenkk/zigbee2mqtt/wiki/How-to-support-new-devices-on-Hass.io

from hassio-zigbee2mqtt.

Hi, I'm having a bit of an issue with the procedure mentioned above at this link. I'm trying to use the custom devices.js in /share/zigbee2mqtt method.

When I enable "zigbee_shepherd_devices": true, the addon starts back up just fine, but when I add an empty devices.js file to the appropriate location I get the following log errors:

> [Info] Configuration file found. Will overwrite configurable fields with values from add-on configuration
> [Info] Configuration written to /share/zigbee2mqtt/configuration.yaml
> [Info] Searching for custom devices file in zigbee2mqtt data path...
> 2018-11-29T08:12:16: PM2 log: Launching in no daemon mode
> 2018-11-29T08:12:17: PM2 log: App [npm:0] starting in -fork mode-
> 2018-11-29T08:12:17: PM2 log: App [npm:0] online
>  [email protected] start /zigbee2mqtt-0.2.0
>  node index.js
> 		Zigbee2mqtt requires node version >=8.11 10, you are running v8.11.4!
>   zigbee2mqtt:info 2018-11-29 08:12:25 Logging to directory: '/share/zigbee2mqtt/log/2018-11-29.08-12-25'
>   zigbee2mqtt:info 2018-11-29 08:12:27 Starting zigbee2mqtt version 0.2.0 (commit #unknown)
>   zigbee2mqtt:info 2018-11-29 08:12:27 Starting zigbee-shepherd
>   zigbee2mqtt:info 2018-11-29 08:12:30 zigbee-shepherd started
>   zigbee2mqtt:info 2018-11-29 08:12:30 Coordinator firmware version: '20180815'
>   zigbee2mqtt:info 2018-11-29 08:12:30 Currently 7 devices are joined:
> /zigbee2mqtt-0.2.0/node_modules/q/q.js:155
>                 throw e;
>                 ^
> TypeError: devices.find is not a function
>     at Object.findByZigbeeModel (/zigbee2mqtt-0.2.0/node_modules/zigbee-shepherd-converters/index.js:7:43)
>     at Controller.getDeviceStartupLogMessage (/zigbee2mqtt-0.2.0/lib/controller.js:153:54)
>     at devices.forEach (/zigbee2mqtt-0.2.0/lib/controller.js:53:42)
>     at Array.forEach (<anonymous>)
>     at zigbee.start (/zigbee2mqtt-0.2.0/lib/controller.js:52:29)
>     at shepherd.start (/zigbee2mqtt-0.2.0/lib/zigbee.js:59:17)
>     at /zigbee2mqtt-0.2.0/node_modules/q/q.js:2055:17
>     at runSingle (/zigbee2mqtt-0.2.0/node_modules/q/q.js:137:13)
>     at flush (/zigbee2mqtt-0.2.0/node_modules/q/q.js:125:13)
>     at _combinedTickCallback (internal/process/next_tick.js:131:7)
> npm
>  ERR! code ELIFECYCLE
> npm ERR! errno 1
> npm ERR! [email protected] start: `node index.js`
> npm ERR! Exit status 1
> npm
>  ERR! 
> npm ERR!
>  Failed at the [email protected] start script.
> npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
> npm ERR! A complete log of this run can be found in:
> npm ERR!
>      /root/.npm/_logs/2018-11-29T13_12_30_331Z-debug.log
> 2018-11-29T08:12:31: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
> 2018-11-29T08:12:31: PM2 log: App [npm:0] starting in -fork mode-
> 2018-11-29T08:12:31: PM2 log: App [npm:0] online
>  [email protected] start /zigbee2mqtt-0.2.0
>  node index.js

When I try adding this device template,

{
    zigbeeModel: ['Z01-A19NAE26'], // The model ID from: Device with modelID 'Z01-A19NAE26' is not supported.
    model: 'Z01-A19NAE26', // Vendor model number, look on the device for a model number
    vendor: 'Sengled', // Vendor of the device (only used for wiki and startup logging)
    description: 'Sengled element plus', // Description of the device, copy from vendor site. (only used for wiki and startup logging)
    supports: 'brightness and color temperature', // Actions this device supports (only used for wiki)
    fromZigbee: [], // We will add this later
    toZigbee: [], // Should be empty, unless device can be controlled (e.g. lights, switches).
},

I get these errors:

> [Info] Configuration file found. Will overwrite configurable               fields with values from add-on configuration
> [Info] Configuration written to /share/zigbee2mqtt/configuration.yaml
> [Info] Searching for custom devices file in zigbee2mqtt data path...
> 2018-11-29T08:15:20: PM2 log: Launching in no daemon mode
> 2018-11-29T08:15:21: PM2 log: App [npm:0] starting in -fork mode-
> 2018-11-29T08:15:22: PM2 log: App [npm:0] online
>  [email protected] start /zigbee2mqtt-0.2.0
>  node index.js
> 		Zigbee2mqtt requires node version >=8.11 10, you are running v8.11.4!
>   zigbee2mqtt:info 2018-11-29 08:15:29 Logging to directory: '/share/zigbee2mqtt/log/2018-11-29.08-15-29'
> /zigbee2mqtt-0.2.0/node_modules/zigbee-shepherd-converters/devices.js:3
>     model: 'Z01-A19NAE26', // Vendor model number, look on the device for a model number
>          ^
> SyntaxError: Unexpected token :
>     at createScript (vm.js:80:10)
>     at Object.runInThisContext (vm.js:139:10)
>     at Module._compile (module.js:616:28)
>     at Object.Module._extensions..js (module.js:663:10)
>     at Module.load (module.js:565:32)
>     at tryModuleLoad (module.js:505:12)
>     at Function.Module._load (module.js:497:3)
>     at Module.require (module.js:596:17)
>     at require (internal/module.js:11:18)
>     at Object.<anonymous> (/zigbee2mqtt-0.2.0/node_modules/zigbee-shepherd-converters/index.js:3:17)
> npm
>  ERR! code ELIFECYCLE
> npm ERR! errno 1
> npm
>  ERR! [email protected] start: `node index.js`
> npm ERR!
>  Exit status 1
> npm ERR! 
> npm ERR! Failed at the [email protected] start script.
> npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
> npm ERR! A complete log of this run can be found in:
> npm ERR!
>      /root/.npm/_logs/2018-11-29T13_15_31_196Z-debug.log
> 2018-11-29T08:15:31: PM2 log: App [npm:0] exited with code [1] via signal [SIGINT]
> 2018-11-29T08:15:31: PM2 log: App [npm:0] starting in -fork mode-
> 2018-11-29T08:15:31: PM2 log: App [npm:0] online
>  [email protected] start /zigbee2mqtt-0.2.0
>  node index.js

I'm not sure how to continue, and would appreciate any troubleshooting help anyone would be able to send my way.

from hassio-zigbee2mqtt.

Ahh, that makes sense in retrospect. I was under the assumption that the /share/zigbee2mqtt/devices.js entry was appended into the docker devices.js.

In case anyone else runs into this, the default devices.js file can be found here - https://github.com/Koenkk/zigbee-shepherd-converters/blob/master/devices.js

from hassio-zigbee2mqtt.

Good idea overall. I’m inclined to just add options for these as well, implemented the same way the current option is for the most part.

Not sure why we’d need to copy the files out to data/zigbee2mqtt by default. Maybe I’m misunderstanding, but seems like an unnecessary pollution of the data dir for most while basically only saving the step of creating a file (and copying its contents) for the most advanced users.

Big picture, I still say it’d be nice to make these extensible within zigbee2mqtt, but I understand that’s a low priority

from hassio-zigbee2mqtt.

I thought of it like that to avoid duplicates, but in practice it would still give the same issues when updating.

However, the way I view the docker container, is a box where logic can run separately from the rest of the system. Config files however are not always placed in them. I think Hass.io also works like this, the /config folder is placed outside of the container, so it persists between updates. Following that logic, it would be NPM stuff in the container. Maybe the split would be between zigbee2mqtt and zigbee-shepherd-coverters?

from hassio-zigbee2mqtt.

@danielwelch I recommend we close this old issue as the support for devices.js is complete and works with both stable and edge. If there are requests, new issues should be opened for converters (https://github.com/Koenkk/zigbee-herdsman-converters/tree/master/converters)

Later edit: It seems that devices.js method might have broken. There is a new implementation that probably allows more flexibility: Koenkk/zigbee-herdsman-converters#1343

Related issues: #373 #383

Later edit2: There is support for this using built in options from z2m: external_converters. I've added documentation for this in #394

from hassio-zigbee2mqtt.