RFD 107: Discussion about rfd HOT 25 OPEN

After giving it some thought I agree with you @rmustacc.

You are talking about a scenario where one interface can have one or many IPv4 addresses as well as one or many IPv6 addresses.

For example the value passed to VMAPI would be:

[
  { "ipv4_uuid": "72a9cd7d-2a0d-4f45-8fa5-f092a3654ce2", 
    "ipv4_ips": ["192.168.1.234"],
    "ipv6_uuid": "d1516824-ece0-46d0-bbe1-87a120268d16",
    "ipv6_count": 1
  }
]

My thoughts were that we would know the type of network passed into VMAPI so specifying ipvX_network was not necessary. However, in that scenario it would be impossible to provision with the above scenario. I think the api interface should be updated to more closely reflect VMAPI's interface.

from rfd.

Theres some background on primaryIP and primary NIC in this ticket:
https://smartos.org/bugview/PUBAPI-1291

Perhaps @marsell has more input on what that should actually look like since I believe he is currently making some changes in that area.

from rfd.

When comparing the flexibility we have/want on the VMAPI end, particularly with IPv6 coming in the future, is there a reason that we're not mirroring some of flexibility of the 'xxx_count' and 'xxx_ips' options?

Particularly, how do you expect this to change when we want to allow CloudAPI to specify both an IPv4 and IPv6 network on a single NIC?

from rfd.

@rmustacc any WIP or completed RFDs @papertigers should cite? (I'm curious about the have vs. want space in VMAPI.)

from rfd.

@danmcd This was all done before RFDs existed, but @papertigers is already citing the docs in VMAPI which talk about this.

from rfd.

@papertigers You make a good point though. There's no good reason to really overburden the user and figure out if it's an IPv4 or IPv6 network. The main reason I wanted to mention it was to make sure that we could specify multiple networks / nic and specify the array of IPs and count. It may make sense to just have the fact that it's IPv4 / IPv6 be figured out inside of Triton.

At least, on the surface, it seems like an unnecessary piece of information to force a user to know, especially when the simple case of only one network is going to be the primary thing.

from rfd.

Some notes from discussion today:

• List IPs
• Get IP
  • Allocated
  • Reserved
  • Instance using it
• This should work for networks that users are owners on (both fabrics / VLANs)
• This should maybe even work for public networks for the IPs that one has allocated (ListIPs / Get IP)
• How does backwards compat with the existing CreateVM bits?
• Known addresses that should be reserved and can't be specified:
  • Broadcast
  • Network
• How does setting the primary NIC fit in here?

from rfd.

It'll be worthwhile to have a section on the key use cases to help us to visualize the workflow and supporting API calls involved. Here are some examples I can think of:

1. Pre-allocated IP ranges for different applications/environments/teams
   Sometimes a subnet is shared between Dev/Test. The way to maintain a soft quota is to assign a range of IP addresses to each of them. In this case, there is no need to create and reserve the nic objects ahead of time. A specific IP address will be passed during provisioning requests. When instances are destroyed, the nics are destroyed along with them. What users in this case will also need are:

• ability to see which IP addresses in the subnet are in use / free, and which instances are using them
• validation against the request for an IP address that is already in use

2. Designated IP addresses for applications that are "bound" to them
   Some application licenses are linked to IP addresses. Some have IP-based firewall rules or DNS mappings. There is a practical need to keep their IP addresses unchanged to avoid the overhead of maintaining these dependencies. As such, these fixed IP addresses need to be reserved at all times. When the applications using them are de-provisioned and provisioned again, the IP addresses should not be grabbed by another instance. Users in this case will need:

• ability to reserve an IP address
• ability to keep an IP address reserved after an instance is destroyed or the nic is removed from the instance

3. Similar to 2, there are also cases when fixed mac addresses are required
   NIC objects with a specific mac and optionally a specific IP may be created and reserved ahead of time. End user in this case will need to be able to pass the mac address in the NIC request.

from rfd.

Working on pushing out an updated version addressing the above feedback.

I think that if we do end up allowing users to specify mac addresses it will only be the last three octets so that our OUI stays the same. I will leave that out of the first update I am going to push and we can discuss that after.

from rfd.

Changes pushed.

Thanks @melloc and @askfongjojo for the feedback on vmapi changes.

This should maybe even work for public networks for the IPs that one has allocated (ListIPs / Get IP)

Does napi currently let you query a public network by account uuid, and is there a way to make sure the GetIP call is being called only on an IP you own?

How does setting the primary NIC fit in here?

Still unclear. For now I have added it to AddNic, but that means will also probably want an UpdateNic.

It'll be worthwhile to have a section on the key use cases to help us to visualize the workflow and supporting API calls involved.

Let me get that section written up.

The question about being able to specify the mac address is still an open issue. Still not totally convinced we should allow this, and if we do it should only be the last three octets.

from rfd.

For AddNIC, if we get to a world with multiple IPs on a NIC, how does a "boolean" for primary help? Shouldn't it indicate a specific IP? Or is just knowing the NIC is primary sufficent?

from rfd.

Relevant code-change in the above ticket: TritonDataCenter/sdc-cloudapi@2a296b0 (They acknowledge a future where multiple IPs (v4 or v6) can be on a single NIC.)

from rfd.

Thanks for the updates. Based on what's here I have a few follow up
questions:

• In the AddNic section, it sounds like the main thing that we're
  allowing now is to specify the ability to add an 'ip' entry to specify
  the specific IP, is that right?

• Should we define what kinds of errors we're going to get back if we try
  to provision an IP on a network that we don't own, but can provision.
  For example, on the public networks? We may want to define the set of
  CloudAPI level errors for all the modified / new entry points.

• Can we more explicitly state which Networks one can list? Can I
  perform a ListNetworkIPs of a newtork I have something provisioned on
  like a shared public network, if so how are we going to limit the
  provisionability?

• Can we talk through a few of the lifecycle things here. For example,
  say I have an IP that's provisioned. Can I mark it reserved? If so, what
  happens to it when I delete the instance that's provisioned it. Does it
  stay reserved?

• Should we have fabric aliases for the IP listing capabilities? In
  other words, should GET /:login/fabrics/default/vlans/:vlan_id/networks/:network/ips/:ip_address
  be an alias for GET /:login/networks/:id/ips/:ip_address.

from rfd.

In the AddNic section, it sounds like the main thing that we're
allowing now is to specify the ability to add an 'ip' entry to specify
the specific IP, is that right?

Thats right. What I don't know is if this endpoint should allow you to specify that this ip is also the primaryIp. Anyone have thoughts on this?

Should we define what kinds of errors we're going to get back if we try
to provision an IP on a network that we don't own, but can provision.
For example, on the public networks? We may want to define the set of
CloudAPI level errors for all the modified / new entry points.

Agreed. I will start working on this.
Also, in the 'Future' section of AddNic I provide an example of what adding multiple IP's to a nic looks like. Do we want to define what happens in that case? For example one of the specified ips is not available etc. In that situation its probably best we validate that the entire request can be completed without issue so that we do not end up with a scenario where only part of the request is successful.

Can we more explicitly state which Networks one can list? Can I
perform a ListNetworkIPs of a newtork I have something provisioned on
like a shared public network, if so how are we going to limit the
provisionability?

I think you should be able to list any private network you are an owner of, including one another user has given you access to.

I also think listing IPs on public network owned by your account would be ideal. However, the napi docs say we can filter a networks ips by belongs_to_uuid but not something like owner_uuid. However looking at the source the moray bucket has the info. So the validation schema can probably be updated to allow filtering by owner. @melloc do you have any concerns with this?

Can we talk through a few of the lifecycle things here. For example,
say I have an IP that's provisioned. Can I mark it reserved? If so, what
happens to it when I delete the instance that's provisioned it. Does it
stay reserved?

This is how it worked in my head when I wrote the rfd. The napi docs document similar lifecycle things. I will add it to be explicit.

Should we have fabric aliases for the IP listing capabilities? In
other words, should GET /:login/fabrics/default/vlans/:vlan_id/networks/:network/ips/:ip_address
be an alias for GET /:login/networks/:id/ips/:ip_address.

This was my intention.

from rfd.

Types of NAPI IP objects:

  // 1. I believe, this is an IP that was allocated (e.g. perhaps because of
  //    earlier usage), but is now free.
  {
    "ip": "10.99.99.52",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": false,
    "free": true
  },
  
  // 2. An IP assigned (belonging) to a given server.
  {
    "ip": "10.99.99.45",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": false,
    "free": false,
    "belongs_to_type": "server",
    "belongs_to_uuid": "564dc847-1949-276a-8eb3-8d3a1df000ca",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },
  
  // 3. An IP assigned (belonging) to a given instance.
  {
    "ip": "10.99.99.46",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": false,
    "free": false,
    "belongs_to_type": "zone",
    "belongs_to_uuid": "2f612f25-792d-4d13-9360-44a862cd1cb0",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },
  
  // 4. A *reserved* IP. (It also happens to be assigned to an instance.)
  {
    "ip": "10.99.99.11",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": true,
    "free": false,
    "belongs_to_type": "zone",
    "belongs_to_uuid": "7ce84909-8ebf-4072-963f-64f603c94f34",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },
  
  // 5. A *specially reserved* IP (belongs_to_type=other).
  //
  //    NAPI currently does this (belongs_to_type=other) for
  //    (a) network.gateway, (b) network.resolvers, and (c) IPv4 broadcast
  //    addresses. In these cases the `belongs_to_uuid` is set to either the
  //    UFDS admin user UUID (i.e. owned by the system) or (for a non-NATed
  //    fabric gateway) to '00000000-0000-0000-0000-000000000000'.
  // 
  //    AFAIK, the NAPI API can be used by operators to set
  //    belongs_to_uuid=other for other purposes.
  {
    "ip": "10.99.99.255",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": true,
    "free": false,
    "belongs_to_type": "other",
    "belongs_to_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  }

Re ListNetworkIPs and GetNetworkIP:

• Q1: The current cloudapi patch sets triton_reserved=true for an IP
  with belongs_to_uuid=other to indicate (I'm inferring) "this is reserved by
  the system".

  • Does NAPI always guarantee that reserved=true when belongs_to_uuid=other?
  • Do we want to have cloudapi define a triton_reserved=true property?
  • Having triton_reserved be either true or undefined on CloudAPI IP
    objects is a little weird.

• Q2: Should CloudAPI IP objects expose belongs_to_type? This could be relevant
  for admin/operator users that can see belongs_to_type=server for some cases.
  If not, then should CloudAPI explicitly filter out belongs_to_type=server
  IPs?

from rfd.

Just wanted to dump some of my comments on this from discussion over chat.

It may be worth while to expose all of the raw napi output with exception to the free property, which is defined by the napi docs as follows:

Setting the free property on an IP removes all other properties from the IP (including reserved,
belongs_to_uuid, and belongs_to_type). This therefore makes the IP available for automatic
provisioning again.

Having a belongs_to_uuid=other would be much better than having a made up cloudapi concept aka triton_reserved=true. Especially because having a property either set to true or undefined like @trentm pointed out is a little awkward. We just have to make sure its documented in cloudapi that type other means you can't update that specific ip aka the goal of triton_reserved

I also now realize there is value in including the owner_uuid property. For example when a user is listing ip's on a private network that has multiple owners, it would be nice to know which account owns that instance on that network.

So, in addition to Trent's questions, is the napi interface considered stable? Do we have an issue with returning raw napi output?

cc @rmustacc @melloc @danmcd

from rfd.

My notes from the call. There are some "Q" questions below.

Example NAPI IP object:

  {
    "ip": "10.99.99.255",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": true,
    "free": false,
    "belongs_to_type": "other",
    "belongs_to_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  }

CloudAPI IP objects:

• Only show IPs for:
  (a) networks for which the account is in network.owner_uuids; or
  (b) a public network (no network.owner_uuids) for which the account
  owns the IP (ip.owner_uuid === account.uuid).
  Q: Do I have that last part right? I'm not totally clear of the
  difference in NIC and IP owner_uuid.
• field ip: always show
• field reserved: always show
• field owner_uuid: always show (allows seeing IP is owned by another user that
  is sharing a private network)
• fields belongs_to_type & belongs_to_uuid: Only show if:
  (a) you are the owner (this biases to not showing info on another
  user's VMs; we could add later if we felt it was fine); and
  (b) belongs_to_type==="zone" (we felt that showing belongs_to_type=server
  would probably be fine, but it is rare -- only admin and maybe
  poseidon accounts -- and it is easier to add it later if needed
  rather than remove it later if it is a problem).
• new managed field: A boolean (see name discussion below) that is true if
  belongs_to_type==="other". It should be managed: false for the other
  cases.
  We talked about in the future being able to specify if an IP is managed by
  a router service vs. a DBaaS vs. other things. It was felt that we would
  still want a boolean field, and the "managed by what" could be answered by
  expanding belongs_to_type values in NAPI later to be more specific than
  just "other".

Naming "managed": Something that says "this IP is managed by something other
than you". "You can't release this IP." Ideas:

• managed
• system_managed
• triton_managed
• thesaurus works: administered, controlled, ruled, managed

Q: What does a NAPI IP object for a volume storage zone nic look like? What
should the cloudapi IP object look like for it?

Q: What does a NAPI IP object for a NAT zone NIC look like?

from rfd.

Q: What does a NAPI IP object for a NAT zone NIC look like?

A nat zone in nightly-1 has a vm.nics[1] like this:

    {
      "interface": "net1",
      "mac": "90:b8:d0:1e:ef:b3",
      "vlan_id": 2,
      "nic_tag": "sdc_overlay/8227494",
      "gateway": "192.168.128.1",
      "gateways": [
        "192.168.128.1"
      ],
      "netmask": "255.255.252.0",
      "ip": "192.168.128.1",
      "ips": [
        "192.168.128.1/22"
      ],
      "network_uuid": "b0427f14-89aa-419b-b84a-c3de86b88e0c",
      "allow_ip_spoofing": true,
      "mtu": 8500
    }

and its NAPI IP object is the first one here:

[root@headnode (nightly-1) ~]# sdc-napi /networks/b0427f14-89aa-419b-b84a-c3de86b88e0c/ips
[
  {
    "ip": "192.168.128.1",
    "network_uuid": "b0427f14-89aa-419b-b84a-c3de86b88e0c",
    "reserved": true,
    "free": false,
    "belongs_to_type": "zone",
    "belongs_to_uuid": "18329628-2b5a-4809-affa-458f7ce4fe7a",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },
...
  {
    "ip": "192.168.131.255",
    "network_uuid": "b0427f14-89aa-419b-b84a-c3de86b88e0c",
    "reserved": true,
    "free": false,
    "belongs_to_type": "other",
    "belongs_to_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  }
]

Following the rules mentioned above, the CloudAPI IP object for that NAT zone's
IP would be:

  {
    "ip": "192.168.128.1",
    "reserved": true,
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "managed": false
  },

which isn't quite right yet because we'd want "managed": true I think.

Q: So perhaps the managed rule needs to be updated to be true if
belongs_to_type === "other" or the owner_uuid is the "admin" account?

ip.managed = (napiIp.belongs_to_type==="other" || napiIp.owner_uuid === adminUuid);

from rfd.

The NAT zone's napi payload is what I was trying to get at on the call, so thanks for grabbing that output.

field owner_uuid: always show (allows seeing IP is owned by another user that
is sharing a private network)

When an IP is reserved by a user or operator napi will show it as follows:

  {
    "ip": "192.168.128.4",
    "network_uuid": "080db52e-8ccf-4a6a-9176-56b746345db3",
    "reserved": true,
    "free": false
  }

In this case there is no owner_uuid field to relay back to cloudapi but users still need to be able to see reserved IPs.

Q: In the case above do you want "owner_uuid": "" or would you rather the field be omitted?

from rfd.

@papertigers

When an IP is reserved by a user or operator napi will show it as follows:

Should that example show reserved=true?

from rfd.

@trentm it should have, you might have seen github before I edited the comment with the right paste.

from rfd.

We plan on exposing owner_uuid in all ip's returned when its set. However, things like a users nat zone is owned by the admin user.

Q. Do we have concerns over exposing the admin UUID?
We could always go with a uuid of all zero's or something like the string "triton", although thats not a valid uuid.

from rfd.

Q. Do we have concerns over exposing the admin UUID?
We could always go with a uuid of all zero's or something like the string "triton", although thats not a valid uuid.

As an operator I would very much prefer all zero’s or anything else over exposing the admin UUID if possible, just to be on the safe side.

from rfd.

Q. Do we have concerns over exposing the admin UUID?
We could always go with a uuid of all zero's or something like the string "triton", although thats not a valid uuid.

The decision from chat was that we should omit owner_uuid in this case rather than lie about it.

from rfd.

FWIW we already expose the admin UUID as the "owner" field on public and admin-owned images. That could theoretically also be changed in future versions of the APIs. I'm still fine with us starting by omitting owner_uuid if it is the admin UUID.

Q: In the case above do you want "owner_uuid": "" or would you rather the field be omitted?

@papertigers I'd rather owner_uuid be omitted if the field is not present on the NAPI IP object.

from rfd.