Git Product home page Git Product logo

ergw's Introduction

erGW - 3GPP GGSN and PDN-GW in Erlang

Build Status Coverage Status Erlang Versions

This is a 3GPP GGSN and PDN-GW implemented in Erlang. It strives to eventually support all the functionality as defined by 3GPP TS 23.002 Section 4.1.3.1 for the GGSN and Section 4.1.4.2.2 for the PDN-GW.

CONTENTS

IMPLEMENTED FEATURES

Messages:

  • GTPv1 Create/Update/Delete PDP Context Request on Gn
  • GTPv2 Create/Delete Session Request on S5/S8

From the above the following procedures as defined by 3GPP T 23.060 should work:

  • PDP Context Activation/Modification/Deactivation Procedure
  • PDP Context Activation/Modification/Deactivation Procedure using S4
  • Intersystem Change Procedures (handover 2G/3G/LTE)
  • 3GPP TS 23.401:
    • Sect. 5.4.2.2, HSS Initiated Subscribed QoS Modification (without PCRF)
    • Annex D, Interoperation with Gn/Gp SGSNs procedures (see CONFIG.md)

EXPERIMENTAL FEATURES

Experimental features may change or be removed at any moment. Configuration settings for them are not guaranteed to work across versions. Check CONFIG.md and NEWS.md on version upgrades.

  • rate limiting, defaults to 100 requests/second
  • metrics, see METRICS.md

USER PLANE

erGW uses the 3GPP control and user plane separation (CUPS) of EPC nodes architecture as layed out in 3GPP TS 23.214 and 3GPP TS 29.244.

DIAMETER and RADIUS over Gi/SGi

The SAE-GW, PGW and GGSN interfaces supports DIAMETER and RADIUS over the Gi/SGi interface as specified by 3GPP TS 29.061 Section 16. This support is experimental in this version and not all aspects are functional. For RADIUS only the Authentication and Authorization is full working, Accounting is experimental and not fully supported. For DIAMETER NASREQ only the Accounting is working.

See RADIUS.md for a list of supported Attrbiutes.

Many thanks to On Waves for sponsoring the RADIUS Authentication implementation.

Example of configuration RADIUS:

%% ...
{ergw_aaa, [
    {handlers, [
        {ergw_aaa_static, [
            {'Node-Id',        <<"CHANGE-ME">>},            %% <- CHANGE
            {'NAS-Identifier', <<"CHANGE-ME">>},            %% <- CHANGE
            {'NAS-IP-Address', {127,0,0,3}},                %% <- CHANGE
            {'Acct-Interim-Interval',   1800},              %% <- CHANGE
            {'Framed-Protocol',         'PPP'},
            {'Service-Type',            'Framed-User'}
        ]},
        {ergw_aaa_radius, [
            {server,
                {{127,0,0,4}, 1813, <<"CHANGE-ME-SECRET">>} %% <- CHANGE IP and SECRET
            },
            {termination_cause_mapping, [
                {normal, 1},
                {administrative, 6},
                {link_broken, 2},
                {upf_failure, 9},
                {remote_failure, 9},
                {cp_inactivity_timeout, 4},
                {up_inactivity_timeout, 4},
                {'ASR', 6},
                {error, 9},
                {peer_restart, 7}
            ]}
        ]}
    ]},
    {services, [
        {'Default', [
            {handler, 'ergw_aaa_static'}
        ]},
        {'RADIUS-Acct', [
            {handler, 'ergw_aaa_radius'}
        ]}
    ]},
    {apps, [
        {default, [
            {session, ['Default']},
            {procedures, [
                {authenticate, []},
                {authorize, []},
                {start, ['RADIUS-Acct']},
                {interim, ['RADIUS-Acct']},
                {stop, ['RADIUS-Acct']}
            ]}
        ]}
    ]}
]},
%% ...

Example of configuration epc-ocs function of DIAMETER:

%% ...
{ergw_aaa, [
%% ...
    {functions, [
        {'epc-ocs', [
            {handler, ergw_aaa_diameter},
            {'Origin-Host', <<"CHANGE-ME">>},                           %% <- CHANGE: Origin-Host needs to be resolvable 
                                                                        %% to local IP (either through /etc/hosts or DNS)
            {'Origin-Realm', <<"CHANGE-ME">>},                          %% <- CHANGE
            {transports, [
                [
                    {connect_to, <<"aaa://CHANGE-ME;transport=tcp">>},  %% <- CHANGE
                    {recbuf,131072},                                    %% <- CHANGE
                    {sndbuf,131072}                                     %% <- CHANGE
                ]
            ]}
        ]}
    ]},
%% ...
]},
%% ...

Example of configuration ergw-pgw-epc-rf function of DIAMETER:

%% ...
{ergw_aaa, [
    %% ...
    {functions, [
        {'ergw-pgw-epc-rf', [
            {handler, ergw_aaa_diameter},
            {'Origin-Host', <<"CHANGE-ME">>},                           %% <- CHANGE
            {'Origin-Realm', <<"CHANGE-ME">>},                          %% <- CHANGE
            {transports, [
                [
                    {connect_to, <<"aaa://CHANGE-ME;transport=tcp">>},  %% <- CHANGE
                    {recbuf,131072},                                    %% <- CHANGE
                    {reuseaddr,false},                                  %% <- CHANGE
                    {sndbuf,131072}                                     %% <- CHANGE
                ]
            ]}
        ]},
    ]},
    {handlers, [
        %% ...
        {ergw_aaa_rf, [
            {function, 'ergw-pgw-epc-rf'},
            {'Destination-Realm', <<"CHANGE-ME">>}                      %% <- CHANGE
        ]},
        {termination_cause_mapping, [
            {normal, 1},           
            {administrative, 4}, 
            {link_broken, 5},      
            {upf_failure, 5},      
            {remote_failure, 1},   
            {cp_inactivity_timeout, 4},
            {up_inactivity_timeout, 4},
            {'ASR', 6},
            {error, 9},
            {peer_restart, 1} 
        ]}
        %% ...
    ]},
    {services, [
        %% ...
        {'Rf', [{handler, 'ergw_aaa_rf'}]},
        %% ...
    ]},
    {apps, [
        {default, [
            %% ...
            {procedures, [
                %% ...
                { {rf, 'Initial'}, ['Rf']},
                { {rf, 'Update'}, ['Rf']},
                { {rf, 'Terminate'}, ['Rf']},
                %% ...
            ]}
        ]}
        %% ...
    ]}
]},
%% ...

POLICY CONTROL

DIAMETER is Gx is supported as experimental feature. Only Credit-Control-Request/Answer (CCR/CCA) and Abort-Session-Request/Answer (ASR/ASA) procedures are supported. Re-Auth-Request/Re-Auth-Answer (RAR/RAA) procedures are not supported.

ONLINE/OFFLINE CHARING

Online charging through Gy is in beta quality with the following known caveats:

  • When multiple rating groups are in use, CCR Update requests will contain unit reservation requests for all rating groups, however they should only contain the entries for the rating groups where new quotas, threshold and validity's are needed.

Offline charging through Rf is supported in beta quality in this version and works only in "independent online and offline charging" mode (tight interworking of online and offline charging is not supported).

Like on Gx only CCR/CCR and ASR/ASA procredures are supported.

MISSING FEATURES

The following procedures are assumed/known to be NOT working:

  • Secondary PDP Context Activation Procedure
  • Secondary PDP Context Activation Procedure using S4

Other shortcomings:

  • QoS parameters are hard-coded

ERLANG Version Support

All minor version of the current major release and the highest minor version of the previous major release will be supported. Due to a bug in OTP 22.x, the netdev configuration option of erGW is broken (see). If you need this feature, you must use OTP 23.x.

When in doubt check the otp_release section in .github/workflows/main.yml for tested versions.

DOCKER IMAGES

Docker images are build by GitHub Actions and pushed to hub.docker.com, and by gitlab.com and pushed to quay.io.

BUILDING DOCKER IMAGE

erGW Docker image can be get from quay.io. For create a new image based on ergw-c-node from quay.io need run second command:

$ docker run -t -i --rm quay.io/travelping/ergw-c-node:2.4.2 -- /bin/sh
/ # cd opt
/opt # ls
ergw-c-node

BUILDING & RUNNING

REQUIRED

  • Erlang OTP 23.2.7 is the recommended version.
  • Rebar3 An erGW installation needs a user plane provider to handle the GTP-U path. This instance can be installed on the same or different host.

A suitable user plane node based on VPP can be found at VPP-UFP.

CONFIGURATION

erGW can be started with rebar3 command line tools, and build with run can looks like:

$ git clone https://github.com/travelping/ergw.git
$ cd ergw
$ wget https://s3.amazonaws.com/rebar3/rebar3
$ chmod u+x ./rebar3
$ touch ergw.config

Then fill just created ergw.config file with content like described below providing a suitable configuration, e.g.:

%-*-Erlang-*-
[{setup, [{data_dir, "/var/lib/ergw"},
          {log_dir,  "/var/log/ergw-c-node"}
         ]},

 {kernel,
  [{logger,
    [{handler, default, logger_std_h,
      #{level => info,
        config =>
            #{sync_mode_qlen => 10000,
              drop_mode_qlen => 10000,
              flush_qlen     => 10000}
       }
     }
    ]}
  ]},

 {ergw, [{'$setup_vars',
          [{"ORIGIN", {value, "epc.mnc001.mcc001.3gppnetwork.org"}}]},
         {plmn_id, {<<"001">>, <<"01">>}},

         {http_api,
          [{port, 8080},
           {ip, {0,0,0,0}}
          ]},

         {node_id, <<"pgw.$ORIGIN">>},
         {sockets,
          [{cp, [{type, 'gtp-u'},
             {vrf, cp},
             {ip,  {127,0,0,1}},
             freebind,
             {reuseaddr, true}
            ]},
           {irx, [{type, 'gtp-c'},
                  {vrf, epc},
                  {ip,  {127,0,0,1}},
                  {reuseaddr, true}
                 ]},
           {sx, [{type, 'pfcp'},
                 {socket, cp},
                 {ip,  {172,21,16,2}}
           ]}
          ]},

         {vrfs,
          [{sgi, [{pools,  [{{10, 106, 0, 1}, {10, 106, 255, 254}, 32},
                            {{16#8001, 0, 0, 0, 0, 0, 0, 0},
                             {16#8001, 0, 0, 16#FFFF, 0, 0, 0, 0}, 64}
                           ]},
                  {'MS-Primary-DNS-Server', {8,8,8,8}},
                  {'MS-Secondary-DNS-Server', {8,8,4,4}},
                  {'MS-Primary-NBNS-Server', {127,0,0,1}},
                  {'MS-Secondary-NBNS-Server', {127,0,0,1}}
                 ]}
          ]},

         {handlers,
          [{'h1', [{handler, pgw_s5s8},
                   {protocol, gn},
                   {sockets, [irx]},
                   {node_selection, [default]}
                  ]},
           {'h2', [{handler, pgw_s5s8},
                   {protocol, s5s8},
                   {sockets, [irx]},
                   {node_selection, [default]}
                  ]}
          ]},

         {apns,
          [{[<<"tpip">>, <<"net">>], [{vrf, sgi}]},
           {[<<"APN1">>], [{vrf, sgi}]}
          ]},

         {teid, {3, 6}}, % {teid, {Prefix, Length}} - optional, default: {0, 0}

         {metrics, [
             {gtp_path_rtt_millisecond_intervals, [10, 100]} % optional, default: [10, 30, 50, 75, 100, 1000, 2000]
         ]},

         {node_selection,
          [{default,
            {static,
             [
              %% APN NAPTR alternative
              {"_default.apn.$ORIGIN", {300,64536},
               [{"x-3gpp-upf","x-sxb"}],
               "topon.sx.prox01.$ORIGIN"},

              %% A/AAAA record alternatives
              {"topon.sx.prox01.$ORIGIN", [{127,0,0,1}], []}
             ]
            }
           }
          ]
         },

         {nodes,
          [{default,
            [{vrfs,
              [{cp, [{features, ['CP-Function']}]},
               {epc, [{features, ['Access']}]},
               {sgi, [{features, ['SGi-LAN']}]}]
             },
             {heartbeat, [
               {interval, 5000},
               {timeout, 500},
               {retry, 5}
             ]},
             {request,
               [{timeout, 30000},
               {retry, 5}]}]
           }]
         },

         {path_management, [
           {t3, 10000},
           {n3,  5},
           {echo, 60000},
           {idle_timeout, 1800000},
           {idle_echo,     600000},
           {down_timeout, 3600000},
           {down_echo,     600000},
           {icmp_error_handling, immediate} % optional, can be 'ignore' | 'immediate', by default: immediate
         ]}
        ]},

 {ergw_aaa,
  [{handlers,
    [{ergw_aaa_static,
        [{'NAS-Identifier',          <<"NAS-Identifier">>},
         {'Acct-Interim-Interval',   600},
         {'Framed-Protocol',         'PPP'},
         {'Service-Type',            'Framed-User'},
         {'Node-Id',                 <<"PGW-001">>},
         {'Charging-Rule-Base-Name', <<"cr-01">>},
         {rules, #{'Default' =>
                       #{'Rating-Group' => [3000],
                         'Flow-Information' =>
                             [#{'Flow-Description' => [<<"permit out ip from any to assigned">>],
                                'Flow-Direction'   => [1]    %% DownLink
                               },
                              #{'Flow-Description' => [<<"permit out ip from any to assigned">>],
                                'Flow-Direction'   => [2]    %% UpLink
                               }],
                         'Metering-Method'  => [1],
                         'Precedence' => [100]
                        }
                  }
         }
        ]}
    ]},

   {services,
    [{'Default', [{handler, 'ergw_aaa_static'}]}
    ]},

   {apps,
    [{default,
      [{session, ['Default']},
       {procedures, [{authenticate, []},
                     {authorize, []},
                     {start, []},
                     {interim, []},
                     {stop, []}
                    ]}
      ]}
    ]}
  ]},

 {jobs, [{samplers,
          [{cpu_feedback, jobs_sampler_cpu, []}
          ]},
         {queues,
          [{path_restart,
            [{regulators, [{counter, [{limit, 100}]}]},
             {modifiers,  [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
            ]},
           {create,
            [{max_time, 5000}, %% max 5 seconds
             {regulators, [{rate, [{limit, 100}]}]},
             {modifiers,  [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
            ]},
           {delete,
            [{regulators, [{counter, [{limit, 100}]}]},
             {modifiers,  [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
            ]},
           {other,
            [{max_time, 10000}, %% max 10 seconds
             {regulators, [{rate, [{limit, 1000}]}]},
             {modifiers,  [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
            ]}
          ]}
        ]}
].

COMPILE & RUN

$ ./rebar3 compile
$ sudo ./rebar3 shell --setcookie secret --sname ergw --config ergw.config --apps ergw
===> Verifying dependencies...
CONFIG: enabling persistent_term support
===> Analyzing applications...
===> Compiling ergw
Erlang/OTP 23 [erts-11.0.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V11.0.3  (abort with ^G)
(ergw@localhost)1> application:info().

The configuration is documented in CONFIG.md

RUNNING UNIT TEST

Unit test can be run local with:

$ rebar ct

In order to run the IPv6 a number of locap IPv6 addresses have to be added to the host. Check .github/workflows/main.yml or .gitlab-ci.yml the list.

The DNS resolver tests can be run with a local DNS server. The docker image use with the CI test can also be use for that.

Run it with:

docker run -d --rm \
        --name=bind9 \
        --publish 127.0.10.1:53:53/udp \
        --publish 127.0.10.1:53:53/tcp \
        --publish 127.0.10.1:953:953/tcp \
        quay.io/travelping/ergw-dns-test-server:latest

and

export CI_DNS_SERVER=127.0.10.1

before running the unit tests.

ergw's People

Contributors

0xax avatar ebengt avatar fholzhauser avatar hwinkel avatar mgumz avatar platinumthinker avatar roadrunnr avatar surik avatar tlnd avatar vkatsuba avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ergw's Issues

DPDK Support

Has anyone considered using dpdk as a data path provider or VPP, or OVS?

rework context/bearer handling to be reference point agnostic

Currently contexts are bound to a process that serves a protocol specific reference point and the protocol can no be changed.

A compliant P-GW implementation needs to support moving context from S5/S8 to Gn/Gp and vice versa (see 3GPP TS 23.401 Annex D (normative): Interoperation with Gn/Gp SGSNs).

Currently, during a handover, the new protocol handler would not know the TEI and simply drop this and any further message. This leaves the old tunnel dangling (since no future messages will arrive for it).

We therefore need to support both GTPv1 and GTPv2 in a P-GW implementation.
Implementation options:

  • decouple context from v1/v2 specific processes and only execute changes to them in protocol specific call back handlers, or
  • send and convert context states on protocol changes from the old handler process to a newly instantiated process

crash during first start

Hi,

I am trying to build/start ergw for the first time, and getting wierd crash.
Have no any idea how to address it, sorry if this issue isn't properly raised.

config:

[{setup, [{data_dir, "/var/lib/ergw"},
          {log_dir,  "/var/log/gtp-c-node"}                             %% NOTE: lager is not using this
         ]},

 {ergw, [{sockets,
          [{irx, [{type, 'gtp-c'},
                  {ip,  {10,254,1,205}},
                  {netdev, "eth"},
                  freebind
                 ]},
           {grx, [{type, 'gtp-u'},
                  {node, 'gtp-u-node@localhost'},
                  {name, 'grx'}]}
          ]},

         {vrfs,
          [{upstream, [{pools,  [{{10, 180, 0, 1}, {10, 180, 255, 254}, 32} ]},
                       {'MS-Primary-DNS-Server', {8,8,8,8}},
                       {'MS-Secondary-DNS-Server', {8,8,4,4}},
                       {'MS-Primary-NBNS-Server', {127,0,0,1}},
                       {'MS-Secondary-NBNS-Server', {127,0,0,1}}
                      ]}
          ]},

         {handlers,
          [{gn, [{handler, pgw_s5s8},
                 {sockets, [irx]},
                 {data_paths, [grx]},
                 {aaa, [{'Username',
                         [{default, ['IMSI', '@', 'APN']}]}]}
                ]},
           {s5s8, [{handler, pgw_s5s8},
                   {sockets, [irx]},
                   {data_paths, [grx]}
                  ]}
          ]},

         {apns,
          [{[<<"example">>, <<"net">>], [{vrf, upstream}]}
          ]}
        ]},

 {gtp_u_kmod, [
        {sockets, [{grx, [{ip, {10,254,1,205}},
                           {netdev, "grx"},
                           freebind,
                           {vrf, [{routes, [{{10, 180, 0, 0}, 16}]},
                                          {netdev, "upstream"}
                                         ]}
                          ]}
        ]}
 ]},

 {ergw_aaa, [
             {ergw_aaa_provider, {ergw_aaa_mock, [{secret, "MySecret"}]}}
            ]},

 {lager, [
          {log_root, "/var/log/gtp-c-node"},
          {colored, true},
          {handlers, [
                      {lager_console_backend, debug},
                      {lager_file_backend, [{file, "error.log"}, {level, error}]},
                      {lager_file_backend, [{file, "console.log"}, {level, debug}]}
                     ]}
         ]}

].

crash:


  crasher:
    initial call: application_master:init/4
    pid: <0.132.0>
    registered_name: []
    exception exit: {{shutdown,
                         {failed_to_start_child,dict,
                             {function_clause,
                                 [{filename,join,
                                      [{error,bad_name},"dictionary.map"],
                                      [{file,"filename.erl"},{line,410}]},
                                  {eradius_dict,'-do_load_tables/2-fun-0-',2,
                                      [{file,"src/eradius_dict.erl"},
                                       {line,78}]},
                                  {lists,flatmap,2,
                                      [{file,"lists.erl"},{line,1250}]},
                                  {eradius_dict,do_load_tables,2,
                                      [{file,"src/eradius_dict.erl"},
                                       {line,77}]},
                                  {eradius_dict,init,1,
                                      [{file,"src/eradius_dict.erl"},
                                       {line,54}]},
                                  {gen_server,init_it,6,
                                      [{file,"gen_server.erl"},{line,328}]},
                                  {proc_lib,init_p_do_apply,3,
                                      [{file,"proc_lib.erl"},{line,247}]}]}}},
                     {eradius,start,[normal,[]]}}
      in function  application_master:init/4 (application_master.erl, line 134)
    ancestors: [<0.131.0>]
    messages: [{'EXIT',<0.133.0>,normal}]
    links: [<0.131.0>,<0.31.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 27
    reductions: 142
  neighbours:

Broken log output

We see logs with special characters instead of the UE's IPv4 address

foo-cnode 06:06:20.139 [error] Sx Node TDF Report on <<6,97,99,99,101,115,115>> for UE IPv4 <<"\n๏ฟฝ๏ฟฝ๏ฟฝ">> IPv6 undefined
foo-cnode 06:06:20.467 [error] Sx Node TDF Report on <<6,97,99,99,101,115,115>> for UE IPv4 <<"\n๏ฟฝ๏ฟฝ๏ฟฝ">> IPv6 undefined
foo-cnode 06:06:21.310 [error] Sx Node TDF Report on <<6,97,99,99,101,115,115>> for UE IPv4 <<"\n๏ฟฝ~๏ฟฝ">> IPv6 undefined

GTP session release is missing when data plane experinces idle timeout

erGW version

repository: quay.io/travelping/ergw-c-node
tag: 3.1.0-rc2

Erlang/OTP version

repository: quay.io/travelping/ergw-c-node
tag: 3.1.0-rc2

Steps to reproduce

Data plane gets Idle-Timout of 600000 milliseconds over pfcp. Start session without data. After 10 minutes pfcp message received that session is deleted.

Current behavior

No GTP delete request sent to SGW

Expected behavior

GTP delete request sent to SGW

Config

Extracted Idle-Timeout config from total
{
"aaa":{
"services":[
{
"service":"Default",
"handler":"ergw_aaa_static",
"answers":[
{
"name":"AA-Auth"
"avps":{
"Result-Code":2001,
"Idle-Timeout":600000
}
}
]
}
]
}

No available resources in PDP context creation

Hi @RoadRunnr , I changed the configuration file (from #196 ) as you suggested so that the ip_pools parameter is now fine but a new error showed up in the response for the 'Create PDP Context request' is code 209 (User Authentication failed) because default values using sgsnemu where {mig, hemmelig}. I changed those to match the one accepted by ergw in {ergw, ergw} and again the same error showed up (199, No resource available).

If it can help this is the output error from the erlang shell:

=ERROR REPORT==== 23-Jul-2020::10:32:50.294199 ===
unable to find context {irx,
{{gtp_port,irx,
<<3,101,112,99>>,
'gtp-c',<0.1123.0>,5,
{172,20,20,1}},
{172,20,20,150},
2123,create_pdp_context_request,
{v1,30721}}}

I attach the modified configuration file:

%--Erlang--

[

%% setup application

{setup,
[{data_dir, "/var/lib/ergw"},
{log_dir, "/var/log/ergw-c-node"}
]},

%%kernel application

{kernel,
[{logger,
[{handler, default, logger_std_h,
#{level => all,
config =>
#{sync_mode_qlen => 10000,
drop_mode_qlen => 10000,
flush_qlen => 10000}
}
}
]}
]},

%% ergw application

{ergw,
[
{'$setup_vars',
[{"ORIGIN", {value, "epc.mnc001.mcc001.3gppnetwork.org"}}
]},

%% GTP sockets

 {sockets,
  [{'cp-socket',
    [{type, 'gtp-u'},
     {vrf, cp},
     {ip, {172,20,21,1}},
     {netdev, "sxb-cp"},
     %{netns, "cp"},
     {reuseaddr, true}
    ]},
   {irx, [{type, 'gtp-c'},
	  {vrf, epc},
	  {ip, {172,20,20,1}},
	  {netdev, "grx-cp"},
	  %{netns, "epc"},
	  {reuseaddr, true}
     ]}
  ]},

%% IP pools

 {ip_pools,
	[{'pool-A', [{ranges, [{{10,80,0,1}, {10,80,255,254}, 32}]},
		{'MS-Primary-DNS-Server', {8,8,8,8}},
		{'MS-Secondary-DNS-Server', {8,8,4,4}},
		{'MS-Primary-NBNS-Server', {127,0,0,1}},
		{'MS-Secondary-NBNS-Server', {127,0,0,1}}
	]},
	{'pool-B', [{ranges, [{{10,81,0,1}, {10,81,255,254}, 32}]},
		{'MS-Primary-DNS-Server', {8,8,8,8}},
		{'MS-Secondary-DNS-Server', {8,8,4,4}},
		{'MS-Primary-NBNS-Server', {127,0,0,1}},
		{'MS-Secondary-NBNS-Server', {127,0,0,1}}
	]},
	{'pool-C', [{ranges, [{{10,82,0,1}, {10,82,255,254}, 32}]},
		{'MS-Primary-DNS-Server', {8,8,8,8}},
		{'MS-Secondary-DNS-Server', {8,8,4,4}},
		{'MS-Primary-NBNS-Server', {127,0,0,1}},
		{'MS-Secondary-NBNS-Server', {127,0,0,1}}
	]}
]},

%% Socket for Sx interface

 {sx_socket,
  [{node, 'ergw'},
   {name, 'ergw'},
   {socket, 'cp-socket'},
   {ip, {172,20,21,1}},
   {netdev, "sxb-cp"},
   {reuseaddr, true}
  ]},

% Interface handlers

 {handlers,
  [{'h1', [{handler, pgw_s5s8},
	   {protocol, gn},
	   {sockets, [irx]},
	   {node_selection, [default]},
	   {aaa, [{'Username',
			[{default, ['IMSI', <<"/">>,
				    'IMEI', <<"/">>,
				    'MSISDN', <<"/">>,
				    'ATOM', <<"/">>,
				    'TEXT', <<"/">>,
				    12345,
				    <<"@">>, 'APN']}]}]}
	  ]},
   {'h2', [{handler, pgw_s5s8},
	   {protocol, s5s8},
	   {sockets, [irx]},
	   {node_selection, [default]},
	   {aaa, [{'Username',
			[{default, ['IMSI', <<"/">>,
				    'IMEI', <<"/">>,
				    'MSISDN', <<"/">>,
				    'ATOM', <<"/">>,
				    'TEXT', <<"/">>,
				    12345,
				    <<"@">>, 'APN']}]}]}
	  ]}
  ]},

%% Network APNs

 {apns,
  [{[<<"tpip">>, <<"net">>], 
	[{vrf, sgi},
	{ip_pools, ['pool-A']},
	{prefered_bearer_type, 'IPv4'}
	]},
   {[<<"APN1">>], 
	[{vrf, sgi},
	{ip_pools, ['pool-A']},
	{prefered_bearer_type, 'IPv4'}
	]}
  ]},

 {node_selection,
  [{default,
    {static,
     [
      %% APN NAPTR alternative
      {"_default.apn.$ORIGIN", {300,64536},
	[{"x-3gpp-pgw", "x-s5-gtp"}, {"x-3gpp-pgw", "x-s8-gtp"}, {"x-3gpp-pgw", "x-gn"}, {"x-3gpp-pgw", "x-gp"}],
	"topon.s5s8.pgw.$ORIGIN"},

      {"_default.apn.$ORIGIN", {300,64536},
       [{"x-3gpp-upf","x-sxb"}],
       "topon.sx.prox01.$ORIGIN"},

      {"web.apn.$ORIGIN", {300,64536},
       [{"x-3gpp-upf","x-sxb"}],
       "topon.sx.prox01.$ORIGIN"},

      %% A/AAAA record alternatives
      {"topon.s5s8.pgw.$ORIGIN", [{172,20,20,150}], []},

      {"topon.sx.prox01.$ORIGIN", [{172,20,21,2}], []}
     ]
    }
   }
  ]
 },

 {nodes,
  [{default,
    [{vrfs,
      [{cp, [{features, ['CP-Function']}]},
       {epc, [{features, ['Access']}]},
       {sgi, [{features, ['SGi-LAN']}]}
     ]},
     {ip_pools, ['pool-A']}
  ]}%%,
  %%{"topon.sx.prox01.$ORIGIN", [connect]}
]}
]},

%% {exometer_core, [{reporters, [{exometer_report_netdata, []}]}]},

%% ergw_aaa application

{ergw_aaa,
[{transports, []},
{handlers,
[{ergw_aaa_static,
[{'NAS-Identifier', <<"NAS-Identifier">>},
{'Node-Id', <<"PGW-001">>},
{'Charging-Rule-Base-Name', <<"m2m0001">>}
]}
]},

{services,
[{'Default', [{handler, 'ergw_aaa_static'}]}]},

{apps,
[{default,
[{session, ['Default']},
{procedures, [{authenticate, []},
{authorize, []},
{start, []},
{interim, []},
{stop, []}
]}
]}
]}
]}
].

Fix Swagger GET metrics

erGW version

2.8.4

Description

Swagger API for get prometheus metrics provide error 404. Looks like the issue in wrong path to endpoint and in wrong accept Content-Type in Swagger config.

Current behavior

When try to call endpoint for get prometheus metrics will be provide error 404 instead of expected metrics data.

Expected behavior

Swagger API for get prometheus metrics should return data of metrics.

Feature: Configurable retry timings for SMF / UPF heartbeats

Between SMF and UPF continuous heartbeat requests are sent to detect availability of the associated UPFs. If these heartbeat requests are not replied in a given time frame, the UPF is considered to be unavailable and thus all the sessions are released.

The timeout value at the moment is hardcoded and rather strict:

ergw/src/ergw_sx_node.erl

Lines 643 to 647 in 09e97cb

send_heartbeat(#data{dp = #node{ip = IP}}) ->
IEs = [#recovery_time_stamp{
time = ergw_gsn_lib:seconds_to_sntp_time(gtp_config:get_start_time())}],
Req = #pfcp{version = v1, type = heartbeat_request, ie = IEs},
ergw_sx_socket:call(IP, 500, 5, Req, response_cb(heartbeat)).

5s for the first timeout
0.5 time out for retries
5 retries

Which makes it like 7.5s to deduce the UPF is not reachable.

For some deployment scenarios that is a really tight deadline to meet.

With a configurable values the situation could be adjusted to the concrete deployment scenario.

VRF Routing not working

Hi

When testing from sgsn simulator, I try to ping different endpoints after establishing PDP context. I am able to ping the host on which the ergw runs (10.0.0.151, ens3). I am able to ping upstream vrf (10.0.0.155 ens8).

I am not able to ping the default gateway or anything beyond. My config below. My instinct says that the routing is not correctly instantiated:

vrf-lite: iproute2 version iproute2-ss161212
#!/bin/sh

ifup() {
/sbin/ip link add $1 type vrf table $2
/sbin/ip link set dev $1 up
/sbin/ip rule add oif $1 table $2
/sbin/ip rule add iif $1 table $2

/sbin/ip link set dev $3 master $1
/sbin/ip link set dev $3 up
/sbin/ip addr flush dev $3
/sbin/ip addr add $4 dev $3
/sbin/ip route add table $2 default via $5

}

ifup upstream 10 ens8 10.0.0.155/24 10.0.0.1
ifup grx 20 ens9 10.0.0.154/24 10.0.0.1

exit 0

ergw:
{vrfs,
[{upstream, [{pools, [{{10, 0, 0, 1}, {10, 0, 0, 254}, 24},
{{16#8001, 0, 0, 0, 0, 0, 0, 0}, {16#8001, 0, 0, 16#FFFF, 0, 0, 0, 0}, 64}
]},
{'MS-Primary-DNS-Server', {8,8,8,8}},
{'MS-Secondary-DNS-Server', {8,8,4,4}},
{'MS-Primary-NBNS-Server', {127,0,0,1}},
{'MS-Secondary-NBNS-Server', {127,0,0,1}}
]}
]},

gtp_u_kmod:
{gtp_u_kmod, [
{sockets, [{grx, [{ip, {10,0,0,154}},
{netdev, "grx"},
{vrf, [{routes, [{{10, 0, 0, 0}, 24}]},
{netdev, "upstream"}
]}
]}
]}
]},

After starting all processes the route command shows:
root@ggsn:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 0 0 0 ens3
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 ens3

Unable to allocate IP pools

Hi,

I'm testing the deployment of ergw PGW + vpp UPF with the addition of a sgsnemu node. The nodes run without errors but the 'Create PDP context request' initiated by the SGSN receives negative response from the PGW with code 199 (No resources available) after the 'Sx Association Setup' and 'Sx session establishment'.

Appearently, inspecting the sniffed packets, the PGW is unable to allocate IP addresses for the end users although I defined the IP pools in the ergw.config file for building the release.

I'm wondering what am I doing wrong, down here I attach the configuration file for ergw:

%--Erlang--

[{setup, [{data_dir, "/var/lib/ergw"},
{log_dir, "/var/log/ergw-c-node"}
]},

{kernel,
[{logger,
[{handler, default, logger_std_h,
#{level => info,
config =>
#{sync_mode_qlen => 10000,
drop_mode_qlen => 10000,
flush_qlen => 10000}
}
}
]}
]},

{ergw,

    [{'$setup_vars',
  [{"ORIGIN", {value, "epc.mnc001.mcc001.3gppnetwork.org"}}]},
 {http_api,
  [{port, 8080},
   {ip, {0,0,0,0}}
  ]},
 {sockets,
  [{'cp-socket',
    [{type, 'gtp-u'},
     {vrf, cp},
     {ip, {172,20,21,1}},
     {netdev, "sxb-cp"},
     {reuseaddr, true}
    ]},
   {irx, [{type, 'gtp-c'},
	  {vrf, epc},
	  {ip, {172,20,20,1}},
	  {netdev, "grx-cp"},
	  {reuseaddr, true}
	 ]}
  ]},

 {vrfs,
  [{sgi, [{pools,  [{{10, 80, 0, 1}, {10, 80, 255, 254}, 32},
		    {{16#8001, 0, 0, 0, 0, 0, 0, 0},
		     {16#8001, 0, 0, 16#FFFF, 0, 0, 0, 0}, 64}
		   ]},
	  {'MS-Primary-DNS-Server', {8,8,8,8}},
	  {'MS-Secondary-DNS-Server', {8,8,4,4}},
	  {'MS-Primary-NBNS-Server', {127,0,0,1}},
	  {'MS-Secondary-NBNS-Server', {127,0,0,1}}
	 ]}
  ]},

 {sx_socket,
  [{node, 'ergw'},
   {name, 'ergw'},
   {socket, 'cp-socket'},
   {ip, {172,20,21,1}},
   {netdev, "sxb-cp"}
  ]},

 {handlers,
  [{'h1', [{handler, pgw_s5s8},
	   {protocol, gn},
	   {sockets, [irx]},
	   {node_selection, [default]}
	  ]},
   {'h2', [{handler, pgw_s5s8},
	   {protocol, s5s8},
	   {sockets, [irx]},
	   {node_selection, [default]}
	  ]}
  ]},

 {apns,
  [{[<<"tpip">>, <<"net">>], [{vrf, sgi}]},
   {[<<"APN1">>], [{vrf, sgi}]}
  ]},

 {node_selection,
  [{default,
    {static,
     [
      %% APN NAPTR alternative
      {"_default.apn.$ORIGIN", {300,64536},
       [{"x-3gpp-upf","x-sxb"}],
       "topon.sx.prox01.$ORIGIN"},

      {"web.apn.$ORIGIN", {300,64536},
       [{"x-3gpp-upf","x-sxb"}],
       "topon.sx.prox01.$ORIGIN"},

      %% A/AAAA record alternatives
      {"topon.sx.prox01.$ORIGIN", [{172,20,21,2}], []}
     ]
    }
   }
  ]
 },

 {nodes,
  [{default,
    [{vrfs,
      [{cp, [{features, ['CP-Function']}]},
       {epc, [{features, ['Access']}]},
       {sgi, [{features, ['SGi-LAN']}]}]
     }]
   }]
 }
]},

{ergw_aaa,
[{transports, []},
{handlers,
[{ergw_aaa_static,
[{'NAS-Identifier', <<"NAS-Identifier">>},
{'Framed-Protocol', 'PPP'},
{'Service-Type', 'Framed-User'},
{'Node-Id', <<"PGW-001">>},
{'Charging-Rule-Base-Name', <<"m2m0001">>},
{rules, #{'Default' =>
#{'Rating-Group' => [3000],
'Flow-Information' =>
[#{'Flow-Description' => [<<"permit out ip from any to assigned">>],
'Flow-Direction' => [1] %% DownLink
},
#{'Flow-Description' => [<<"permit out ip from any to assigned">>],
'Flow-Direction' => [2] %% UpLink
}],
'Metering-Method' => [1],
'Precedence' => [100]
}
}
}
]}
]},

{services,
[{'Default', [{handler, 'ergw_aaa_static'}]}]},

{apps,
[{default,
[{session, ['Default']},
{procedures, [{authenticate, []},
{authorize, []},
{start, []},
{interim, []},
{stop, []}
]}
]}
]}
]},

{hackney, [
{mod_metrics, exometer}
]},

{jobs, [{samplers,
[{cpu_feedback, jobs_sampler_cpu, []}
]},
{queues,
[{path_restart,
[{regulators, [{counter, [{limit, 100}]}]},
{modifiers, [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
]},
{create,
[{max_time, 5000}, %% max 5 seconds
{regulators, [{rate, [{limit, 100}]}]},
{modifiers, [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
]},
{delete,
[{regulators, [{counter, [{limit, 100}]}]},
{modifiers, [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
]},
{other,
[{max_time, 10000}, %% max 10 seconds
{regulators, [{rate, [{limit, 1000}]}]},
{modifiers, [{cpu_feedback, 10}]} %% 10 = % increment by which to modify the limit
]}
]}
]}
].

Unable to run ergw on Ubuntu 16.10

Hi

I have been trying to run ergw on my Ubuntu 16.10 install for about a week now, without success.

I have followed instructions on https://github.com/travelping/ergw, but with no success. I would really appreciate it if I can receive some guidance on running ergw. I think the first thing that I should get working is https://github.com/travelping/gtp_u_kmod or am I on the wrong track? I would really appreciate any assistance. I think that if I get gtp_u_kmod going, I should be able to start ergw.

I include my config:
[
{node, [
{run_as_user, "msupra"},
{run_as_group, "msupra"},
{smp, enabled}
]},

{gtp_u_kmod, [
{sockets, [{grx, [{ip, {192,168,123,74}},
{netdev, "grx"},
{port, 16384},
{routes, [{{10, 180, 0, 0}, 16}]}
]}
]}
]}].

The commands I run:
cd /ggsn/v3/gtp_u_kmod/ebin/
export ERL_LIBS=/ggsn/v3/gtp_u_kmod/deps/
erl -setcookie secret -sname gtp_u_kmod -config /etc/ggsn/gtp.config
application:ensure_all_started(gtp_u_kmod).

The final output below:
`Erlang/OTP 19 [erts-8.2] [source-fbd2db2] [64-bit] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V8.2 (abort with ^G)
(gtp_u_kmod@ggsn)1> application:ensure_all_started(gtp_u_kmod).
14:33:29.464 [info] Application lager started on node gtp_u_kmod@ggsn
14:33:29.464 [info] Application erlando started on node gtp_u_kmod@ggsn
14:33:29.466 [info] Application gen_socket started on node gtp_u_kmod@ggsn
14:33:29.475 [info] Application gen_netlink started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,sasl_safe_sup}
started: [{pid,<0.113.0>},
{id,alarm_handler},
{mfargs,{alarm_handler,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,sasl_sup}
started: [{pid,<0.112.0>},
{id,sasl_safe_sup},
{mfargs,
{supervisor,start_link,
[{local,sasl_safe_sup},sasl,safe]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,sasl_sup}
started: [{pid,<0.114.0>},
{id,release_handler},
{mfargs,{release_handler,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: sasl
started_at: gtp_u_kmod@ggsn
14:33:29.487 [info] Application sasl started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: regine
started_at: gtp_u_kmod@ggsn
14:33:29.488 [info] Application regine started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: crypto
started_at: gtp_u_kmod@ggsn
14:33:29.492 [info] Application crypto started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: hut
started_at: gtp_u_kmod@ggsn
14:33:29.494 [info] Application hut started on node gtp_u_kmod@ggsn
14:33:29.496 [info] Setup running ...
14:33:29.497 [info] Directories verified. Res = ok

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: setup
started_at: gtp_u_kmod@ggsn
14:33:29.497 [info] Setup finished processing hooks ...
14:33:29.498 [info] Application setup started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: bear
started_at: gtp_u_kmod@ggsn
14:33:29.500 [info] Application bear started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,folsom_sup}
started: [{pid,<0.128.0>},
{id,folsom_sample_slide_sup},
{mfargs,{folsom_sample_slide_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,supervisor}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,folsom_sup}
started: [{pid,<0.129.0>},
{id,folsom_meter_timer_server},
{mfargs,{folsom_meter_timer_server,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,folsom_sup}
started: [{pid,<0.130.0>},
{id,folsom_metrics_histogram_ets},
{mfargs,{folsom_metrics_histogram_ets,start_link,[]}},
{restart_type,permanent},
{shutdown,2000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: folsom
started_at: gtp_u_kmod@ggsn
14:33:29.505 [info] Application folsom started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,exometer_core_sup}
started: [{pid,<0.135.0>},
{id,exometer_admin},
{mfargs,{exometer_admin,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,exometer_core_sup}
started: [{pid,<0.136.0>},
{id,exometer_cache},
{mfargs,{exometer_cache,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,exometer_core_sup}
started: [{pid,<0.137.0>},
{id,exometer_report},
{mfargs,{exometer_report,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,exometer_core_sup}
started: [{pid,<0.138.0>},
{id,exometer_report_logger_sup},
{mfargs,{exometer_report_logger_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,supervisor}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,exometer_core_sup}
started: [{pid,<0.139.0>},
{id,exometer_folsom_monitor},
{mfargs,{exometer_folsom_monitor,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,exometer_core_sup}
started: [{pid,<0.140.0>},
{id,exometer_alias},
{mfargs,{exometer_alias,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
14:33:29.518 [info] msg: "Starting reporters with []\n", options: []

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: exometer_core
started_at: gtp_u_kmod@ggsn
14:33:29.519 [info] Application exometer_core started on node gtp_u_kmod@ggsn
14:33:29.531 [info] Loaded RADIUS tables: [dictionary]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_sup}
started: [{pid,<0.145.0>},
{id,dict},
{mfargs,{eradius_dict,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_sup}
started: [{pid,<0.147.0>},
{id,node_mon},
{mfargs,{eradius_node_mon,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_sup}
started: [{pid,<0.148.0>},
{id,counter},
{mfargs,{eradius_counter,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_sup}
started: [{pid,<0.149.0>},
{id,aggregator},
{mfargs,{eradius_counter_aggregator,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_sup}
started: [{pid,<0.150.0>},
{id,radius_log},
{mfargs,{eradius_log,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_server_top_sup}
started: [{pid,<0.152.0>},
{id,sup},
{mfargs,{eradius_server_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_server_top_sup}
started: [{pid,<0.153.0>},
{id,mon},
{mfargs,{eradius_server_mon,start_link,[]}},
{restart_type,permanent},
{shutdown,brutal_kill},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_sup}
started: [{pid,<0.151.0>},
{id,server_top_sup},
{mfargs,{eradius_server_top_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,infinity},
{child_type,supervisor}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_sup}
started: [{pid,<0.154.0>},
{id,client},
{mfargs,{eradius_client,start_link,[]}},
{restart_type,permanent},
{shutdown,500},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
application: eradius
started_at: gtp_u_kmod@ggsn
14:33:29.545 [info] Application eradius started on node gtp_u_kmod@ggsn
14:33:29.566 [info] Loaded RADIUS tables: [dictionary,dictionary_alcatel_sr,dictionary_rfc4679]

=PROGRESS REPORT==== 17-Jan-2017::14:33:29 ===
supervisor: {local,eradius_client_sup}
started: [{pid,<0.162.0>},
{id,72},
{mfargs,
{eradius_client_socket,start,
[undefined,<0.154.0>,72]}},
{restart_type,transient},
{shutdown,brutal_kill},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:44 ===
supervisor: {local,ppp_sup}
started: [{pid,<0.172.0>},
{id,ppp_session},
{mfargs,{ppp_session,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:44 ===
supervisor: {local,ppp_sup}
started: [{pid,<0.173.0>},
{id,ppp_link_sup},
{mfargs,{ppp_link_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,supervisor}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:44 ===
application: ppp
started_at: gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:44 ===
application: gtplib
started_at: gtp_u_kmod@ggsn
14:33:44.591 [info] Application ppp started on node gtp_u_kmod@ggsn
14:33:44.591 [info] Application gtplib started on node gtp_u_kmod@ggsn

=PROGRESS REPORT==== 17-Jan-2017::14:33:44 ===
supervisor: {local,gtp_u_kmod_app_sup}
started: [{pid,<0.179.0>},
{id,gtp_u_kmod_port_sup},
{mfargs,{gtp_u_kmod_port_sup,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,supervisor}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:44 ===
supervisor: {local,gtp_u_kmod_app_sup}
started: [{pid,<0.180.0>},
{id,gtp_u_kmod},
{mfargs,{gtp_u_kmod,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
14:33:44.595 [info] RegName: port_grx
{ok,[syntax_tools,compiler,goldrush,lager,erlando,
gen_socket,gen_netlink,sasl,regine,crypto,hut,setup,bear,
folsom,exometer_core,eradius,ppp,gtplib,gtp_u_kmod]}

**=CRASH REPORT==== 17-Jan-2017::14:33:44 ===
crasher:
initial call: gtp_u_kmod_port:init/1
pid: <0.181.0>
registered_name: []
exception exit: {{badmatch,{error,enodev}},
[{gtp_u_kmod_port,socket_setopts,2,
[{file,"src/gtp_u_kmod_port.erl"},{line,169}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
{gtp_u_kmod_port,bind_gtp_socket,4,
[{file,"src/gtp_u_kmod_port.erl"},{line,160}]},
{gtp_u_kmod_port,init,1,
[{file,"src/gtp_u_kmod_port.erl"},{line,78}]},
{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,328}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
in function gen_server:init_it/6 (gen_server.erl, line 352)
ancestors: [gtp_u_kmod_port_sup,gtp_u_kmod_app_sup,<0.177.0>]
messages: []
links: [<0.179.0>,#Port<0.4726>]
dictionary: []
trap_exit: false
status: running
heap_size: 987
stack_size: 27
reductions: 790
neighbours:
(gtp_u_kmod@ggsn)2>
=SUPERVISOR REPORT==== 17-Jan-2017::14:33:44 ===
Supervisor: {local,gtp_u_kmod_port_sup}
Context: child_terminated
Reason: {{badmatch,{error,enodev}},
[{gtp_u_kmod_port,socket_setopts,2,
[{file,"src/gtp_u_kmod_port.erl"},
{line,169}]},
{lists,foreach,2,[{file,"lists.erl"},{line,1338}]},
{gtp_u_kmod_port,bind_gtp_socket,4,
[{file,"src/gtp_u_kmod_port.erl"},
{line,160}]},
{gtp_u_kmod_port,init,1,
[{file,"src/gtp_u_kmod_port.erl"},
{line,78}]},
{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,247}]}]}
Offender: [{pid,<0.181.0>},
{id,gtp_u_kmod_port},
{mfargs,{gtp_u_kmod_port,start_link,undefined}},
{restart_type,temporary},
{shutdown,1000},
{child_type,worker}]

=PROGRESS REPORT==== 17-Jan-2017::14:33:44 ===
application: gtp_u_kmod
started_at: gtp_u_kmod@ggsn
14:33:44.598 [error] CRASH REPORT Process <0.181.0> with 0 neighbours exited with reason: no match of right hand value {error,enodev} in gtp_u_kmod_port:socket_setopts/2 line 169 in gen_server:init_it/6 line 352
14:33:44.600 [error] Supervisor gtp_u_kmod_port_sup had child gtp_u_kmod_port started with {gtp_u_kmod_port,start_link,undefined} at <0.181.0> exit with reason no match of right hand value {error,enodev} in gtp_u_kmod_port:socket_setopts/2 line 169 in context child_terminated
14:33:44.601 [info] Application gtp_u_kmod started on node gtp_u_kmod@ggsn**`

Regards
Morne

UP VTIME feature support

We currently use a timer in the PFCP code to support quota validity times. With Rel.16.4 a new UP feature was introduced to handle this in the UP.

Making use of this feature clears another obstacle on the path to a stateless ergw.

VTIME is optional in the UP, the C-Node therefore needs a fallback. The existing timer server support from exp/stateless branch (once it is sufficiently stable) should be used a basis. Then the support for optionally pushing the timer into the URRs needs to be added.

Fix DNS cache test

erGW version

2.8.4

Description

The DNS cache test case is unstable, the timeout condition does not work in about half of the test runs.
A failure of this test also cascades into the following test. That should be investigated as well.

Current behavior

The DNS cache test case is unstable.

Expected behavior

The DNS cache test case should be stable.

Incorrect default for 'Idle-Timeout'

erGW version

2.3.0

Erlang/OTP version

23.2.7

Description

Optional functionality behind Idle-Timeout setting on the APN level has a default value of 8h, what makes it non optional by default.
It is set in

{'Idle-Timeout', 28800000} %% 8hrs timer in msecs

Current behavior

If nothing is provided in the config - code will default to linked above 8h.

Expected behavior

  • it is optional functionality - so it should default to infinity
  • or at least this should match any of our use cases (if we are using this variable - we are setting there 12h always)

No retry of PFCP association to UPF after first failed try

Description

In a GTP-Proxy setup with real DNS based node selection it was observed, that after selecting a configured upf the PFCP association (or more specific) the "FAR installation procedure" did not succeed.

ergw was configured potentially have cp, epc1, epc2, access1 and access2 as NWIs while the selected *upf was configured to support cp, epc1, epc2and access1. access2 was not supported.

The installation of the FARs after the successfull PFCP association request failed (because upf did not support a NWI required by ergw).

After reconfiguring the upf, restarting it: ergw would not trigger a single PFCP association request to the upf

Side question: ergw wants to install FARs for all available NWIs in the node section. At the same time, the socket-configuration for a access-side socket states it will use only one specific NWI. This is by design? Because ergw can't know upon PFCP establishment which of the NWIs will be used later on?

Relevant configurations

ergw

{ergw, [
    {sockets, [
      {'grx-0', [
        {type, 'gtp-c'},
        {vrf, access1},
        {ip, {192,168,0,1}}
    ]},

    {nodes,
      [{default,
        [{vrfs,
          [{cp, [{features, ['CP-Function']}]},
           {access1, [{features, ['Access']}]},
           {access2, [{features, ['Access']}]},
           {epc1, [{features, ['Core']}]},
           {epc2, [{features, ['Core']}]}
         ]}]

upf

upf nwi name access1 vrf 2
upf gtpu endpoint ip 192.168.0.2 nwi access1 teid 0x80000000/2

upf nwi name epc1 vrf 3
upf gtpu endpoint ip 127.0.0.3 nwi epc1 teid 0x80000000/2

upf nwi name epc2 vrf 4
upf gtpu endpoint ip 127.0.0.4 nwi epc2 teid 0x80000000/2

Versions:

  • travelping/ergw:2.4.3
  • travelping/vpp:v20.09-upfbase2

Add missing decription of PFCP metrics

Need to update METRICS.md and provide description of PFCP metrics below:

Name Description
pfcp_errors_total ...
pfcp_messages_duplicates_total ...
pfcp_messages_processed_total ...
pfcp_messages_replies_total ...
pfcp_messages_retransmits_total ...
pfcp_messages_timeouts_total ...
pfcp_peer_response_milliseconds bucket, count, sum
pfcp_request_duration_microseconds bucket, count, sum

Feature: Allow multiples DNS servers to perform queries for Node Selection

Context

ergw can use either a static, internal DNS like setup to select the UPF nodes (or SMF nodes in a GTP Proxy setup) or a real DNS server. The configuration for the DNS server looks like this:

{node_selection,
  [
    {'dnsid', {dns, {{192,168,0,53}, 53} }},

This allows to use only one DNS server.

Envisioned Feature:

In order to achieve redundancy, ergw should be able to use at least a 2nd DNS server.

Remarks:

The config validation code accepts only one DNS server at the moment

validate_options(dns, IP) when ?IS_IP(IP) ->
{IP, 53};
validate_options(dns, {IP, Port} = Server)
when ?IS_IP(IP) andalso is_integer(Port) ->
Server;

The ergw_node_selection:lookup_host function makes use of a list of nameservers:

lookup_host(Name, dns, NameServers) ->
NsOpts =
case NameServers of
{_,_} ->
[{nameservers, [NameServers]}];
_ ->
[]
end,
%% 3GPP DNS answers are almost always large, use TCP by default....
case inet_res:resolve(Name, in, any, [{usevc, true} | NsOpts]) of

naptr(Name, NameServers) ->
NsOpts =
case NameServers of
{_,_} ->
[{nameservers, [NameServers]}];
_ ->
[]
end,
%% 3GPP DNS answers are almost always large, use TCP by default....
inet_res:resolve(Name, in, naptr, [{usevc, true} | NsOpts]).

Add metric for active peers

Please add a metric for the current number of active peers.

We currently generate that metric by filtering out peers that sent an echo request in the last X minutes.

Tetrapak build fails

From a fresh git checkout when trying to build ergw I get the following error:

== build:erlang ================== Compiling src/ergw_sup.erl src/ergw_sup.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_socket_sup.erl src/gtp_socket_sup.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_c_lib.erl src/gtp_c_lib.erl: Error: undefined parse transform 'lager_transform' Compiling src/itu_e212.erl src/itu_e212.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_proxy_ds.erl src/gtp_proxy_ds.erl: Error: undefined parse transform 'lager_transform' Compiling src/vrf.erl src/vrf.erl: Error: undefined parse transform 'lager_transform' Compiling src/vrf_reg.erl src/vrf_reg.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_path.erl src/gtp_path.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_api.erl src/gtp_api.erl: Error: undefined parse transform 'lager_transform' Compiling src/pgw_s5s8.erl src/pgw_s5s8.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_context_sup.erl src/gtp_context_sup.erl: Error: undefined parse transform 'lager_transform' Compiling src/exometer_report_netdata.erl src/exometer_report_netdata.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_config.erl src/gtp_config.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_path_reg.erl src/gtp_path_reg.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_path_sup.erl src/gtp_path_sup.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_context.erl src/gtp_context.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_socket.erl src/gtp_socket.erl: Error: undefined parse transform 'lager_transform' Compiling src/ergw_config.erl src/ergw_config.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_context_reg.erl src/gtp_context_reg.erl: Error: undefined parse transform 'lager_transform' Compiling src/ergw.erl src/ergw.erl: Error: undefined parse transform 'lager_transform' Compiling src/ergw_api.erl src/ergw_api.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_protocol.erl src/gtp_protocol.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_v1_u.erl src/gtp_v1_u.erl: Error: undefined parse transform 'lager_transform' Compiling src/vrf_sup.erl src/vrf_sup.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_dp.erl src/gtp_dp.erl: Error: undefined parse transform 'lager_transform' Compiling src/ergw_app.erl src/ergw_app.erl: Error: undefined parse transform 'lager_transform' Compiling src/pgw_s2a.erl src/pgw_s2a.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_v1_c.erl src/gtp_v1_c.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_v2_c.erl src/gtp_v2_c.erl: Error: undefined parse transform 'lager_transform' Compiling src/3gpp_qos.erl src/3gpp_qos.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_socket_reg.erl src/gtp_socket_reg.erl: Error: undefined parse transform 'lager_transform' Compiling src/ggsn_gn_proxy.erl src/ggsn_gn_proxy.erl: Error: undefined parse transform 'lager_transform' Compiling src/pgw_s5s8_proxy.erl src/pgw_s5s8_proxy.erl: Error: undefined parse transform 'lager_transform' Compiling src/ergw_control.erl src/ergw_control.erl: Error: undefined parse transform 'lager_transform' Compiling src/gtp_ip_pool.erl src/gtp_ip_pool.erl: Error: undefined parse transform 'lager_transform' Compiling src/ggsn_gn.erl src/ggsn_gn.erl: Error: undefined parse transform 'lager_transform' Error: compilation failed
When I put lager_transform first in tetrapak/config.ini:
erlc_options = [{lager_transform, parse_transform}]
I get the following:

== build:erlang ================== Compiling src/ergw_sup.erl Compiling src/gtp_socket_sup.erl Compiling src/gtp_c_lib.erl src/gtp_c_lib.erl: Error: undefined parse transform 'cut' Compiling src/itu_e212.erl Compiling src/gtp_proxy_ds.erl Compiling src/vrf.erl src/vrf.erl: Error: undefined parse transform 'cut' Compiling src/vrf_reg.erl src/vrf_reg.erl:10: Warning: behaviour regine_server undefined Compiling src/gtp_path.erl src/gtp_path.erl: Error: undefined parse transform 'cut' Compiling src/gtp_api.erl src/gtp_api.erl:3: Error: can't find include lib "gtplib/include/gtp_packet.hrl" src/gtp_api.erl:32: Error: record gtp undefined src/gtp_api.erl:41: Error: record gtp undefined src/gtp_api.erl:42: Error: record gtp undefined Compiling src/pgw_s5s8.erl src/pgw_s5s8.erl: Error: undefined parse transform 'do' Compiling src/gtp_context_sup.erl Compiling src/exometer_report_netdata.erl src/exometer_report_netdata.erl:17: Error: can't find include lib "exometer_core/include/exometer.hrl" src/exometer_report_netdata.erl:71: Error: record exometer_entry undefined src/exometer_report_netdata.erl:72: Error: variable 'PortName' is unbound src/exometer_report_netdata.erl:77: Error: variable 'Name' is unbound src/exometer_report_netdata.erl:77: Error: variable 'Value' is unbound src/exometer_report_netdata.erl:3: Warning: behaviour exometer_report undefined Compiling src/gtp_config.erl Compiling src/gtp_path_reg.erl src/gtp_path_reg.erl:10: Warning: behaviour regine_server undefined Compiling src/gtp_path_sup.erl Compiling src/gtp_context.erl src/gtp_context.erl: Error: undefined parse transform 'cut' Compiling src/gtp_socket.erl src/gtp_socket.erl: Error: undefined parse transform 'cut' Compiling src/ergw_config.erl src/ergw_config.erl: Error: undefined parse transform 'cut' Compiling src/gtp_context_reg.erl src/gtp_context_reg.erl:10: Warning: behaviour regine_server undefined Compiling src/ergw.erl Compiling src/ergw_api.erl Compiling src/gtp_protocol.erl src/gtp_protocol.erl:3: Error: can't find include lib "gtplib/include/gtp_packet.hrl" src/gtp_protocol.erl:7: Error: record gtp undefined src/gtp_protocol.erl:12: Error: record gtp undefined Compiling src/gtp_v1_u.erl src/gtp_v1_u.erl:10: Warning: behaviour gtp_protocol undefined Compiling src/vrf_sup.erl Compiling src/gtp_dp.erl src/gtp_dp.erl:22: Error: can't find include lib "gen_socket/include/gen_socket.hrl" src/gtp_dp.erl:23: Error: can't find include lib "gtplib/include/gtp_packet.hrl" Compiling src/ergw_app.erl src/ergw_app.erl: Error: undefined parse transform 'cut' Compiling src/pgw_s2a.erl src/pgw_s2a.erl: Error: undefined parse transform 'do' Compiling src/gtp_v1_c.erl src/gtp_v1_c.erl:23: Error: can't find include lib "gtplib/include/gtp_packet.hrl" src/gtp_v1_c.erl:33: Error: record recovery undefined src/gtp_v1_c.erl:34: Error: variable 'RestartCounter' is unbound src/gtp_v1_c.erl:53: Error: record gtp undefined src/gtp_v1_c.erl:56: Error: record gtp undefined src/gtp_v1_c.erl:58: Error: record gtp undefined src/gtp_v1_c.erl:230: Error: record recovery undefined src/gtp_v1_c.erl:232: Error: record recovery undefined src/gtp_v1_c.erl:242: Error: record cause undefined src/gtp_v1_c.erl:244: Error: record cause undefined src/gtp_v1_c.erl:246: Error: record cause undefined src/gtp_v1_c.erl:10: Warning: behaviour gtp_protocol undefined src/gtp_v1_c.erl:55: Warning: variable 'IEs' is unused src/gtp_v1_c.erl:55: Warning: variable 'TEI' is unused src/gtp_v1_c.erl:55: Warning: variable 'Type' is unused src/gtp_v1_c.erl:57: Warning: variable 'IEs' is unused src/gtp_v1_c.erl:57: Warning: variable 'Type' is unused src/gtp_v1_c.erl:166: Warning: function gtp_msg_response/1 is unused src/gtp_v1_c.erl:229: Warning: variable 'RCnt' is unused src/gtp_v1_c.erl:231: Warning: variable 'RCnt' is unused src/gtp_v1_c.erl:234: Warning: function map_reply_ies/1 is unused src/gtp_v1_c.erl:241: Warning: function map_reply_ie/1 is unused Compiling src/gtp_v2_c.erl src/gtp_v2_c.erl:23: Error: can't find include lib "gtplib/include/gtp_packet.hrl" src/gtp_v2_c.erl:33: Error: record v2_recovery undefined src/gtp_v2_c.erl:34: Error: variable 'RestartCounter' is unbound src/gtp_v2_c.erl:54: Error: record gtp undefined src/gtp_v2_c.erl:57: Error: record gtp undefined src/gtp_v2_c.erl:59: Error: record gtp undefined src/gtp_v2_c.erl:210: Error: record gtp undefined src/gtp_v2_c.erl:212: Error: variable 'IfType' is unbound src/gtp_v2_c.erl:247: Error: record v2_recovery undefined src/gtp_v2_c.erl:249: Error: record v2_recovery undefined src/gtp_v2_c.erl:259: Error: record v2_cause undefined src/gtp_v2_c.erl:261: Error: record v2_cause undefined src/gtp_v2_c.erl:263: Error: record v2_cause undefined src/gtp_v2_c.erl:10: Warning: behaviour gtp_protocol undefined src/gtp_v2_c.erl:53: Warning: variable 'IEs' is unused src/gtp_v2_c.erl:56: Warning: variable 'IEs' is unused src/gtp_v2_c.erl:56: Warning: variable 'TEI' is unused src/gtp_v2_c.erl:56: Warning: variable 'Type' is unused src/gtp_v2_c.erl:58: Warning: variable 'IEs' is unused src/gtp_v2_c.erl:58: Warning: variable 'Type' is unused src/gtp_v2_c.erl:186: Warning: function gtp_msg_response/1 is unused src/gtp_v2_c.erl:246: Warning: variable 'RCnt' is unused src/gtp_v2_c.erl:248: Warning: variable 'RCnt' is unused src/gtp_v2_c.erl:251: Warning: function map_reply_ies/1 is unused src/gtp_v2_c.erl:258: Warning: function map_reply_ie/1 is unused Compiling src/3gpp_qos.erl Compiling src/gtp_socket_reg.erl src/gtp_socket_reg.erl:10: Warning: behaviour regine_server undefined Compiling src/ggsn_gn_proxy.erl src/ggsn_gn_proxy.erl: Error: undefined parse transform 'cut' Compiling src/pgw_s5s8_proxy.erl src/pgw_s5s8_proxy.erl: Error: undefined parse transform 'cut' Compiling src/ergw_control.erl Compiling src/gtp_ip_pool.erl Compiling src/ggsn_gn.erl src/ggsn_gn.erl: Error: undefined parse transform 'cut' Error: compilation failed
Looks like it doesn't resolve the dependencies

No RADIUS Acct-Stop when Session Termination Pipeline fails earlier

erGW version

3.1.0-rc5

Erlang/OTP version

unrelated

Steps to reproduce

OCS-Free: disabled
Radius enabled procedures: authenticate, authorize, start, stop
OCS: enabled
OFCS: enabled

Configure OCS server to not answer CCR-Update request and create session with small CC-Time

PGW <- CSR <- SGW
PGW -> Access-Request -> Radius
PGW <- Access-Response <- Radius
PGW -> CCR-Initial -> OCS
PGW <- CCA-Initial <- OCS
PGW -> AR-Start -> OFCS
PGW <- AA-Start <- OFCS
PGW -> Accounting-Request Start -> Radius
PGW -> CSR -> SGW
PGW <- Accounting-Response Start <- Radius

* Stop OCS Server here *

PGW -> CCR-Update -> OCS (No response)

Current behavior

PGW -> DBR (Reactivation requested) -> SGW
PGW <- DBR (Request accepted) <- SGW
PGW -> AR-Stop -> OFCS
PGW <- AR-Stop <- OFCS

No request to radius
badarg error in logs:

2021-09-26T14:19:13.302570+00:00 <0.1552.0> debug: <0.1552.0>: found response: 6: Heartbeat Response(V: v1, SEID: undefined, Seq: 6): [#recovery_time_stamp{time = 3841653269}]
2021-09-26T14:19:13.302635+00:00 <0.1552.0> debug: send_request_reply: {send_req,{198,18,0,21},500,5,<<32,1,0,12,0,0,6,0,0,96,0,4,228,250,255,210>>,{pfcp,v1,heartbeat_request,undefined,6,[{recovery_time_stamp,3841654738}]},undefined,{ergw_sx_node,response,[<0.1555.0>,heartbeat]},-576460735670825553}
2021-09-26T14:19:13.302725+00:00 <0.1555.0> debug: PFCP OK Response: #{recovery_time_stamp => #recovery_time_stamp{time = 3841653269}}
2021-09-26T14:19:13.432612+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: <<32,1,0,12,0,2,92,0,0,96,0,4,228,250,250,21>>
2021-09-26T14:19:13.432747+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: Heartbeat Request(V: v1, SEID: undefined, Seq: 604): [#recovery_time_stamp{time = 3841653269}]
2021-09-26T14:19:13.432806+00:00 <0.1552.0> debug: cast HB request to <0.1555.0>
2021-09-26T14:19:13.851042+00:00 <0.1673.0> debug: handle_info: {timeout,#Ref<0.1541156830.3813933064.235777>,pfcp_timer}
2021-09-26T14:19:13.851185+00:00 <0.1555.0> debug: DP Call {pfcp,v1,session_modification_request,4208255547954808539,undefined,[{query_urr,[{urr_id,4}]}]}
2021-09-26T14:19:13.851234+00:00 <0.1552.0> debug: PrepSend: {send_req,{198,18,0,21},3000,3,undefined,{pfcp,v1,session_modification_request,4208255547954808539,undefined,[{query_urr,[{urr_id,4}]}]},undefined,{ergw_sx_node,response,[<0.1555.0>,{call,{<0.1673.0>,#Ref<0.1541156830.3813933061.237678>}}]},-576460735121654683}
2021-09-26T14:19:13.851332+00:00 <0.1552.0> debug: PrepSend: Session Modification Request(V: v1, SEID: 4208255547954808539, Seq: 7): [#query_urr{group = [#urr_id{id = 4}]}]
2021-09-26T14:19:13.887084+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: <<33,53,0,163,58,102,186,31,227,233,198,219,0,0,7,0,0,19,0,1,1,0,78,0,142,0,81,0,4,0,0,0,4,0,104,0,4,0,0,0,0,0,63,0,3,128,0,0,0,75,0,4,228,250,255,210,0,76,0,4,228,250,255,225,0,66,0,49,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,67,0,4,0,0,0,15,128,3,0,10,72,249,228,250,255,225,226,250,52,0,128,4,0,10,72,249,228,250,255,210,226,248,112,0,128,5,0,10,72,249,228,250,255,225,226,248,112,0>>
2021-09-26T14:19:13.887862+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: Session Modification Response(V: v1, SEID: 4208255547954808539, Seq: 7): [#pfcp_cause{cause = 'Request accepted'}, #usage_report_smr{, group =, #{duration_measurement => #duration_measurement{duration = 15}, end_time => #end_time{time = 3841654753}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654753,fraction = 3808048128}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654753,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 4}, usage_report_trigger =>, #usage_report_trigger{, immer = 1,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 0,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}]
2021-09-26T14:19:13.888986+00:00 <0.1552.0> debug: <0.1552.0>: found response: 7: Session Modification Response(V: v1, SEID: 4208255547954808539, Seq: 7): [#pfcp_cause{cause = 'Request accepted'}, #usage_report_smr{, group =, #{duration_measurement => #duration_measurement{duration = 15}, end_time => #end_time{time = 3841654753}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654753,fraction = 3808048128}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654753,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 4}, usage_report_trigger =>, #usage_report_trigger{, immer = 1,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 0,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}]
2021-09-26T14:19:13.890212+00:00 <0.1552.0> debug: send_request_reply: {send_req,{198,18,0,21},3000,3,<<33,52,0,24,58,102,186,31,227,233,198,219,0,0,7,0,0,77,0,8,0,81,0,4,0,0,0,4>>,{pfcp,v1,session_modification_request,4208255547954808539,7,[{query_urr,[{urr_id,4}]}]},undefined,{ergw_sx_node,response,[<0.1555.0>,{call,{<0.1673.0>,#Ref<0.1541156830.3813933061.237678>}}]},-576460735121654683}
2021-09-26T14:19:13.891276+00:00 <0.1674.0> warning: Ro Charging Key: 3000
2021-09-26T14:19:18.303210+00:00 <0.1555.0> debug: sending heartbeat
2021-09-26T14:19:18.303417+00:00 <0.1552.0> debug: PrepSend: {send_req,{198,18,0,21},500,5,undefined,{pfcp,v1,heartbeat_request,undefined,undefined,[{recovery_time_stamp,3841654738}]},undefined,{ergw_sx_node,response,[<0.1555.0>,heartbeat]},-576460730669483444}
2021-09-26T14:19:18.303624+00:00 <0.1552.0> debug: PrepSend: Heartbeat Request(V: v1, SEID: undefined, Seq: 8): [#recovery_time_stamp{time = 3841654738}]
2021-09-26T14:19:18.304270+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: <<32,2,0,12,0,0,8,0,0,96,0,4,228,250,250,21>>
2021-09-26T14:19:18.304631+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: Heartbeat Response(V: v1, SEID: undefined, Seq: 8): [#recovery_time_stamp{time = 3841653269}]
2021-09-26T14:19:18.304847+00:00 <0.1552.0> debug: <0.1552.0>: found response: 8: Heartbeat Response(V: v1, SEID: undefined, Seq: 8): [#recovery_time_stamp{time = 3841653269}]
2021-09-26T14:19:18.305083+00:00 <0.1552.0> debug: send_request_reply: {send_req,{198,18,0,21},500,5,<<32,1,0,12,0,0,8,0,0,96,0,4,228,250,255,210>>,{pfcp,v1,heartbeat_request,undefined,8,[{recovery_time_stamp,3841654738}]},undefined,{ergw_sx_node,response,[<0.1555.0>,heartbeat]},-576460730669483444}
2021-09-26T14:19:18.305463+00:00 <0.1555.0> debug: PFCP OK Response: #{recovery_time_stamp => #recovery_time_stamp{time = 3841653269}}
2021-09-26T14:19:18.314262+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: <<32,1,0,12,0,2,93,0,0,96,0,4,228,250,250,21>>
2021-09-26T14:19:18.314509+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: Heartbeat Request(V: v1, SEID: undefined, Seq: 605): [#recovery_time_stamp{time = 3841653269}]
2021-09-26T14:19:18.314738+00:00 <0.1552.0> debug: cast HB request to <0.1555.0>
2021-09-26T14:19:18.893214+00:00 <0.1955.0> debug: DIAMETER error in ergw_aaa_ro with: {error,timeout}. Request: ['CCR'|#{'3GPP-IMSI' => [<<"001010123456001">>],'Auth-Application-Id' => 4,'CC-Request-Number' => 1,'CC-Request-Type' => 2,'Destination-Realm' => <<"gy.test.cennso.net">>,'Event-Timestamp' => [{{2021,9,26},{14,19,13}}],'Multiple-Services-Credit-Control' => [#{'Rating-Group' => [3000],'Requested-Service-Unit' => #{},'Used-Service-Unit' => [#{'CC-Input-Octets' => [0],'CC-Output-Octets' => [0],'CC-Time' => [15],'CC-Total-Octets' => [0],'Reporting-Reason' => [4]}]}],'Multiple-Services-Indicator' => [1],'Service-Context-Id' => "[email protected]",'Service-Information' => [#{'PS-Information' => [#{'3GPP-Charging-Id' => [3615480409],'3GPP-GGSN-MCC-MNC' => [<<"001001">>],'3GPP-IMSI-MCC-MNC' => [<<"00101">>],'3GPP-MS-TimeZone' => [<<0,0>>],'3GPP-NSAPI' => [<<"5">>],'3GPP-PDP-Type' => [0],'3GPP-RAT-Type' => [<<6>>],'3GPP-SGSN-MCC-MNC' => [<<"001030">>],'3GPP-Selection-Mode' => [<<"0">>],'Called-Station-Id' => [<<"test.apn.net">>],'Charging-Characteristics-Selection-Mode' => [3],'GGSN-Address' => [{198,18,0,11}],'Node-Id' => [<<"pgw.test.cennso.net">>],'PDN-Connection-Charging-ID' => [3615480409],'PDP-Address' => [{198,20,12,198}],'QoS-Information' => [#{'APN-Aggregate-Max-Bitrate-DL' => [0],'APN-Aggregate-Max-Bitrate-UL' => [0],'Allocation-Retention-Priority' => [#{'Pre-emption-Capability' => [1],'Pre-emption-Vulnerability' => [1],'Priority-Level' => 11}],'Guaranteed-Bitrate-DL' => [0],'Guaranteed-Bitrate-UL' => [0],'Max-Requested-Bandwidth-DL' => [0],'Max-Requested-Bandwidth-UL' => [0],'QoS-Class-Identifier' => "\t"}],'SGSN-Address' => [{198,18,0,1}],'Start-Time' => [{{2021,9,26},{14,18,58}}]}]}],'Session-Id' => <<"bdbdd6bf0640;402750498;2032701184;874096397825">>,'Subscription-Id' => [#{'Subscription-Id-Data' => <<"00496448000000">>,'Subscription-Id-Type' => 0},#{'Subscription-Id-Data' => <<"001010123456001">>,'Subscription-Id-Type' => 1}],'User-Equipment-Info' => [#{'User-Equipment-Info-Type' => 0,'User-Equipment-Info-Value' => <<"0026279000000000">>}],'User-Name' => [<<"ergw">>]}]
2021-09-26T14:19:18.893686+00:00 <0.1674.0> error: CCA Result: {error,timeout}
2021-09-26T14:19:18.893834+00:00 <0.1673.0> debug: SessionEvents: #{'Accounting-Start' => -576460750121697852,'Framed-IPv6-Pool' => <<"sgi1">>,'APN' => [<<"test">>,<<"apn">>,<<"net">>],'Framed-Pool' => <<"sgi1">>,'Service-Type' => 'Framed-User','MS-Primary-NBNS-Server' => <<127,0,0,1>>,'3GPP-RAT-Type' => 6,'3GPP-NSAPI' => 5,'3GPP-Charging-Id' => 3615480409,'Acct-Interim-Interval' => 1800,'3GPP-IMSI' => <<"001010123456001">>,'Node-Id' => <<"pgw.test.cennso.net">>,'NAS-IP-Address' => {198,18,0,11},'Username' => <<"ergw">>,'3GPP-SGSN-Address' => {198,18,0,1},'3GPP-IMSI-MCC-MNC' => {<<"001">>,<<"01">>},'3GPP-PDP-Type' => 'IPv4','GTP-Version' => v2,'Credit-Control-Failure-Handling' => continue,'3GPP-SGSN-UP-Address' => {198,18,1,1},'Called-Station-Id' => <<"test.apn.net">>,'Diameter-Session-Id' => <<"bdbdd6bf0640;402750498;2032701184;874096397825">>,'3GPP-SGSN-MCC-MNC' => {<<"001">>,<<"030">>},'Acct-Authentic' => 'RADIUS','Framed-IP-Address' => {198,20,12,198},'Password' => <<"ergw">>,'Session-Id' => 31909181945741612589014337298759978498,'Multi-Session-Id' => 31909181945741612589014337298759978497,'3GPP-MS-TimeZone' => {0,0},used_credits => [{3000,#{'CC-Input-Octets' => [0],'CC-Output-Octets' => [0],'CC-Time' => [15],'CC-Total-Octets' => [0],'Reporting-Reason' => [4]}}],'3GPP-IMEISV' => <<"0026279000000000">>,'QoS-Information' => #{'APN-Aggregate-Max-Bitrate-DL' => 0,'APN-Aggregate-Max-Bitrate-UL' => 0,'Allocation-Retention-Priority' => #{'Pre-emption-Capability' => 1,'Pre-emption-Vulnerability' => 1,'Priority-Level' => 11},'Guaranteed-Bitrate-DL' => 0,'Guaranteed-Bitrate-UL' => 0,'Max-Requested-Bandwidth-DL' => 0,'Max-Requested-Bandwidth-UL' => 0,'QoS-Class-Identifier' => 9},'CC-Session-Failover' => supported,'3GPP-MSISDN' => <<"00496448000000">>,'MS-Secondary-DNS-Server' => <<8,8,4,4>>,'MS-Primary-DNS-Server' => <<"\b\b\b\b">>,'NAS-Identifier' => <<"pgw">>,'Event-Trigger' => 18,credits => #{3000 => empty},'Calling-Station-Id' => <<"00496448000000">>,'MS-Secondary-NBNS-Server' => <<127,0,0,1>>,'3GPP-GGSN-Address' => {198,18,0,11},'Bearer-Operation' => 1,'3GPP-GGSN-MCC-MNC' => {<<"001">>,<<"001">>},'Framed-Protocol' => 'GPRS-PDP-Context','3GPP-Selection-Mode' => 0,'Offline-Charging-Profile' => #{'Default' => #{'Tariff-Time' => #{'Local-Tariff-Time' => {23,0},'Location' => <<"Etc/UTC">>}}},'Session-Start' => -576460750125564415}, Events: [{stop,{gy,timeout}}]
2021-09-26T14:19:18.894577+00:00 <0.1673.0> debug: <0.1673.0>: gtp_socket send_request to 198.18.0.1({198,18,0,1}):2123: {gtp,v2,delete_bearer_request,11010049,undefined,undefined,[],[{v2_cause,0,reactivation_requested,0,0,0,undefined},{v2_eps_bearer_id,0,5}]}
2021-09-26T14:19:18.897943+00:00 <0.1549.0> debug: handle message: {gtp,{198,18,0,1},2123,{socket,<<"epc">>,'gtp-c',<0.1549.0>},{gtp,v2,delete_bearer_response,3362194550,0,undefined,[],<<2,0,2,0,16,0>>}}
2021-09-26T14:19:18.898142+00:00 <0.1549.0> debug: <0.1549.0>: found response: {v2,0}
2021-09-26T14:19:18.898353+00:00 <0.1673.0> debug: handle gtp response: {gtp,v2,delete_bearer_response,3362194550,0,undefined,[],#{{v2_cause,0} => {v2_cause,0,request_accepted,0,0,0,undefined}}}
2021-09-26T14:19:18.898461+00:00 <0.1555.0> debug: DP Call {pfcp,v1,session_deletion_request,4208255547954808539,undefined,[]}
2021-09-26T14:19:18.898562+00:00 <0.1552.0> debug: PrepSend: {send_req,{198,18,0,21},3000,3,undefined,{pfcp,v1,session_deletion_request,4208255547954808539,undefined,[]},undefined,{ergw_sx_node,response,[<0.1555.0>,{call,{<0.1673.0>,#Ref<0.1541156830.3813933061.237701>}}]},-576460730074339216}
2021-09-26T14:19:18.898696+00:00 <0.1552.0> debug: PrepSend: Session Deletion Request(V: v1, SEID: 4208255547954808539, Seq: 9): []
2021-09-26T14:19:18.899067+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: <<33,55,2,89,58,102,186,31,227,233,198,219,0,0,9,0,0,19,0,1,1,0,79,0,142,0,81,0,4,0,0,0,1,0,104,0,4,0,0,0,0,0,63,0,3,0,8,0,0,75,0,4,228,250,255,210,0,76,0,4,228,250,255,230,0,66,0,49,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,67,0,4,0,0,0,20,128,3,0,10,72,249,228,250,255,230,230,31,200,0,128,4,0,10,72,249,228,250,255,210,226,248,112,0,128,5,0,10,72,249,228,250,255,230,226,248,112,0,0,79,0,142,0,81,0,4,0,0,0,2,0,104,0,4,0,0,0,0,0,63,0,3,0,8,0,0,75,0,4,228,250,255,210,0,76,0,4,228,250,255,230,0,66,0,49,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,67,0,4,0,0,0,20,128,3,0,10,72,249,228,250,255,230,230,31,200,0,128,4,0,10,72,249,228,250,255,210,226,248,112,0,128,5,0,10,72,249,228,250,255,230,226,248,112,0,0,79,0,142,0,81,0,4,0,0,0,3,0,104,0,4,0,0,0,0,0,63,0,3,0,8,0,0,75,0,4,228,250,255,210,0,76,0,4,228,250,255,230,0,66,0,49,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,67,0,4,0,0,0,20,128,3,0,10,72,249,228,250,255,230,230,31,200,0,128,4,0,10,72,249,228,250,255,210,226,248,112,0,128,5,0,10,72,249,228,250,255,230,226,248,112,0,0,79,0,142,0,81,0,4,0,0,0,4,0,104,0,4,0,0,0,1,0,63,0,3,0,8,0,0,75,0,4,228,250,255,225,0,76,0,4,228,250,255,230,0,66,0,49,63,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,67,0,4,0,0,0,5,128,3,0,10,72,249,228,250,255,230,230,31,200,0,128,4,0,10,72,249,228,250,255,225,226,248,112,0,128,5,0,10,72,249,228,250,255,230,226,248,112,0>>
2021-09-26T14:19:18.900021+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: Session Deletion Response(V: v1, SEID: 4208255547954808539, Seq: 9): [#pfcp_cause{cause = 'Request accepted'}, [#usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 20}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 1}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 20}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 2}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 20}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 3}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 5}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654753}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654753,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 1}, urr_id => #urr_id{id = 4}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}]]
2021-09-26T14:19:18.901390+00:00 <0.1552.0> debug: <0.1552.0>: found response: 9: Session Deletion Response(V: v1, SEID: 4208255547954808539, Seq: 9): [#pfcp_cause{cause = 'Request accepted'}, [#usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 20}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 1}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 20}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 2}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 20}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654738}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654738,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 3}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 5}, end_time => #end_time{time = 3841654758}, start_time => #start_time{time = 3841654753}, tp_now => #tp_now{seconds = 3841654758,fraction = 3860842496}, tp_start_time =>, #tp_start_time{seconds = 3841654753,fraction = 3807932416}, tp_stop_time =>, #tp_stop_time{seconds = 3841654758,fraction = 3807932416}, ur_seqn => #ur_seqn{number = 1}, urr_id => #urr_id{id = 4}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 0,uplink = 0,downlink = 0,total_pkts = 0, uplink_pkts = 0,downlink_pkts = 0}}}]]
2021-09-26T14:19:18.902424+00:00 <0.1552.0> debug: send_request_reply: {send_req,{198,18,0,21},3000,3,<<33,54,0,12,58,102,186,31,227,233,198,219,0,0,9,0>>,{pfcp,v1,session_deletion_request,4208255547954808539,9,[]},undefined,{ergw_sx_node,response,[<0.1555.0>,{call,{<0.1673.0>,#Ref<0.1541156830.3813933061.237701>}}]},-576460730074339216}
2021-09-26T14:19:18.902747+00:00 <0.1673.0> debug: GxSessionOpts: #{'Accounting-Start' => -576460750121697852,'Framed-IPv6-Pool' => <<"sgi1">>,'APN' => [<<"test">>,<<"apn">>,<<"net">>],'Framed-Pool' => <<"sgi1">>,'Service-Type' => 'Framed-User','MS-Primary-NBNS-Server' => <<127,0,0,1>>,'3GPP-RAT-Type' => 6,'3GPP-NSAPI' => 5,'3GPP-Charging-Id' => 3615480409,'Acct-Interim-Interval' => 1800,'3GPP-IMSI' => <<"001010123456001">>,'Node-Id' => <<"pgw.test.cennso.net">>,'NAS-IP-Address' => {198,18,0,11},'Username' => <<"ergw">>,'3GPP-SGSN-Address' => {198,18,0,1},'3GPP-IMSI-MCC-MNC' => {<<"001">>,<<"01">>},'3GPP-PDP-Type' => 'IPv4','GTP-Version' => v2,'Credit-Control-Failure-Handling' => continue,'3GPP-SGSN-UP-Address' => {198,18,1,1},'Called-Station-Id' => <<"test.apn.net">>,'Diameter-Session-Id' => <<"bdbdd6bf0640;402750498;2032701184;874096397825">>,'3GPP-SGSN-MCC-MNC' => {<<"001">>,<<"030">>},'Acct-Authentic' => 'RADIUS','Framed-IP-Address' => {198,20,12,198},'Password' => <<"ergw">>,'Session-Id' => 31909181945741612589014337298759978498,'Multi-Session-Id' => 31909181945741612589014337298759978497,'3GPP-MS-TimeZone' => {0,0},used_credits => [{3000,#{'CC-Input-Octets' => [0],'CC-Output-Octets' => [0],'CC-Time' => [15],'CC-Total-Octets' => [0],'Reporting-Reason' => [4]}}],'3GPP-IMEISV' => <<"0026279000000000">>,'QoS-Information' => #{'APN-Aggregate-Max-Bitrate-DL' => 0,'APN-Aggregate-Max-Bitrate-UL' => 0,'Allocation-Retention-Priority' => #{'Pre-emption-Capability' => 1,'Pre-emption-Vulnerability' => 1,'Priority-Level' => 11},'Guaranteed-Bitrate-DL' => 0,'Guaranteed-Bitrate-UL' => 0,'Max-Requested-Bandwidth-DL' => 0,'Max-Requested-Bandwidth-UL' => 0,'QoS-Class-Identifier' => 9},'CC-Session-Failover' => supported,'3GPP-MSISDN' => <<"00496448000000">>,'MS-Secondary-DNS-Server' => <<8,8,4,4>>,'MS-Primary-DNS-Server' => <<"\b\b\b\b">>,'NAS-Identifier' => <<"pgw">>,'Event-Trigger' => 18,credits => #{3000 => empty},'Calling-Station-Id' => <<"00496448000000">>,'MS-Secondary-NBNS-Server' => <<127,0,0,1>>,'3GPP-GGSN-Address' => {198,18,0,11},'Bearer-Operation' => 1,'3GPP-GGSN-MCC-MNC' => {<<"001">>,<<"001">>},'Framed-Protocol' => 'GPRS-PDP-Context','3GPP-Selection-Mode' => 0,'Offline-Charging-Profile' => #{'Default' => #{'Tariff-Time' => #{'Local-Tariff-Time' => {23,0},'Location' => <<"Etc/UTC">>}}},'Session-Start' => -576460750125564415}
2021-09-26T14:19:18.903686+00:00 <0.1674.0> debug: Session Stop: #{'Accounting-Start' => -576460750121697852,'Framed-IPv6-Pool' => <<"sgi1">>,'APN' => [<<"test">>,<<"apn">>,<<"net">>],'Framed-Pool' => <<"sgi1">>,'Service-Type' => 'Framed-User','MS-Primary-NBNS-Server' => <<127,0,0,1>>,'3GPP-RAT-Type' => 6,'3GPP-NSAPI' => 5,'3GPP-Charging-Id' => 3615480409,'Acct-Interim-Interval' => 1800,'3GPP-IMSI' => <<"001010123456001">>,'Node-Id' => <<"pgw.test.cennso.net">>,'OutOctets' => 0,'NAS-IP-Address' => {198,18,0,11},'Username' => <<"ergw">>,'3GPP-SGSN-Address' => {198,18,0,1},'3GPP-IMSI-MCC-MNC' => {<<"001">>,<<"01">>},'3GPP-PDP-Type' => 'IPv4','GTP-Version' => v2,'Credit-Control-Failure-Handling' => continue,'3GPP-SGSN-UP-Address' => {198,18,1,1},'Called-Station-Id' => <<"test.apn.net">>,'Diameter-Session-Id' => <<"bdbdd6bf0640;402750498;2032701184;874096397825">>,'Termination-Cause' => timeout,'3GPP-SGSN-MCC-MNC' => {<<"001">>,<<"030">>},'Acct-Authentic' => 'RADIUS','Framed-IP-Address' => {198,20,12,198},'Password' => <<"ergw">>,'Session-Id' => 31909181945741612589014337298759978498,'Multi-Session-Id' => 31909181945741612589014337298759978497,'3GPP-MS-TimeZone' => {0,0},traffic_data => [#{'3GPP-Charging-Id' => [3615480409],'3GPP-RAT-Type' => [<<6>>],'Accounting-Input-Octets' => [0],'Accounting-Output-Octets' => [0],'Change-Condition' => [0],'Change-Time' => [{{2021,9,26},{14,19,18}}],'QoS-Information' => [#{'APN-Aggregate-Max-Bitrate-DL' => 0,'APN-Aggregate-Max-Bitrate-UL' => 0,'Allocation-Retention-Priority' => #{'Pre-emption-Capability' => 1,'Pre-emption-Vulnerability' => 1,'Priority-Level' => 11},'Guaranteed-Bitrate-DL' => 0,'Guaranteed-Bitrate-UL' => 0,'Max-Requested-Bandwidth-DL' => 0,'Max-Requested-Bandwidth-UL' => 0,'QoS-Class-Identifier' => 9}]}],used_credits => [{3000,#{'CC-Input-Octets' => [0],'CC-Output-Octets' => [0],'CC-Time' => [5],'CC-Total-Octets' => [0],'Reporting-Reason' => [2]}}],'3GPP-IMEISV' => <<"0026279000000000">>,'QoS-Information' => #{'APN-Aggregate-Max-Bitrate-DL' => 0,'APN-Aggregate-Max-Bitrate-UL' => 0,'Allocation-Retention-Priority' => #{'Pre-emption-Capability' => 1,'Pre-emption-Vulnerability' => 1,'Priority-Level' => 11},'Guaranteed-Bitrate-DL' => 0,'Guaranteed-Bitrate-UL' => 0,'Max-Requested-Bandwidth-DL' => 0,'Max-Requested-Bandwidth-UL' => 0,'QoS-Class-Identifier' => 9},'CC-Session-Failover' => supported,'Accounting-Stop' => -576460730070280904,'3GPP-MSISDN' => <<"00496448000000">>,'Acct-Session-Time' => 20,'MS-Secondary-DNS-Server' => <<8,8,4,4>>,'MS-Primary-DNS-Server' => <<"\b\b\b\b">>,'NAS-Identifier' => <<"pgw">>,'Event-Trigger' => 18,credits => #{3000 => empty},service_data => [#{'3GPP-RAT-Type' => [<<6>>],'Accounting-Input-Octets' => [0],'Accounting-Output-Octets' => [0],'Change-Condition' => [0],'Change-Time' => [{{2021,9,26},{14,19,18}}],'QoS-Information' => [#{'APN-Aggregate-Max-Bitrate-DL' => 0,'APN-Aggregate-Max-Bitrate-UL' => 0,'Allocation-Retention-Priority' => #{'Pre-emption-Capability' => 1,'Pre-emption-Vulnerability' => 1,'Priority-Level' => 11},'Guaranteed-Bitrate-DL' => 0,'Guaranteed-Bitrate-UL' => 0,'Max-Requested-Bandwidth-DL' => 0,'Max-Requested-Bandwidth-UL' => 0,'QoS-Class-Identifier' => 9}],'Rating-Group' => [3000],'Time-Usage' => [20]}],'Calling-Station-Id' => <<"00496448000000">>,'MS-Secondary-NBNS-Server' => <<127,0,0,1>>,'3GPP-GGSN-Address' => {198,18,0,11},'InOctets' => 0,'Bearer-Operation' => 1,'3GPP-GGSN-MCC-MNC' => {<<"001">>,<<"001">>},'Framed-Protocol' => 'GPRS-PDP-Context','3GPP-Selection-Mode' => 0,'Offline-Charging-Profile' => #{'Default' => #{'Tariff-Time' => #{'Local-Tariff-Time' => {23,0},'Location' => <<"Etc/UTC">>}}},'Session-Start' => -576460750125564415}
2021-09-26T14:19:18.904381+00:00 <0.1963.0> error: crasher: initial call: ergw_aaa_radius:'-accounting/6-fun-0-'/0, pid: <0.1963.0>, registered_name: [], error: {badarg,[{eradius_lib,encode_value,2,[{file,"/build/_build/default/lib/eradius/src/eradius_lib.erl"},{line,196}]},{eradius_lib,encode_attribute,3,[{file,"/build/_build/default/lib/eradius/src/eradius_lib.erl"},{line,177}]},{eradius_lib,'-encode_attributes/2-fun-0-',3,[{file,"/build/_build/default/lib/eradius/src/eradius_lib.erl"},{line,143}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{eradius_lib,encode_request,1,[{file,"/build/_build/default/lib/eradius/src/eradius_lib.erl"},{line,73}]},{eradius_client,send_request_loop,7,[{file,"/build/_build/default/lib/eradius/src/eradius_client.erl"},{line,246}]},{eradius_client,'-send_request/3-fun-0-',10,[{file,"/build/_build/default/lib/eradius/src/eradius_client.erl"},{line,90}]},{proc_lib,init_p,3,[{file,"proc_lib.erl"},{line,211}]}]}, ancestors: [<0.1674.0>,ergw_aaa_session_sup,ergw_aaa_sup,<0.1280.0>], message_queue_len: 0, messages: [], links: [], dictionary: [], trap_exit: false, status: running, heap_size: 2586, stack_size: 28, reductions: 692; neighbours:
2021-09-26T14:19:18.905290+00:00 <0.1674.0> error: Received DOWN information for <0.1673.0> with info normal
2021-09-26T14:19:22.636465+00:00 <0.1552.0> debug: handle message 198.18.0.21:8805: <<32,1,0,12,0,2,94,0,0,96,0,4,228,250,250,21>>

Expected behavior

Stop request to radius with mapped termination cause "req_timeout"

Config

Implement configurable GTP RTT metrics interval

erGW version

2.8.4

Description

This ticket is for the implementation of configurable GTP path RTT metric intervals. The large majority of GTP path metrics are RTT information. On systems maintaining many GTP paths, this can result in such large amount of data that the prometheus metrics collector clients can time out.

Current behavior

The current implementation uses 7 intervals [10, 30, 50, 75, 100, 1000, 2000] which results in 10 entries for each GTP path GTP version and sent_message type.

Expected behavior

An optional configuration option in the ergw is available to specify the intervals in milliseconds to be used for the GTP path RTT metrics. If the configuration option is not defined, it should fall back to the current implemented intervals.

Example of the new configuration option:

{gtp_path_rtt_metric_intervals, [10, 100]}

Do not return hardcoded Bearer QoS in Create Session Response

erGW version

2.8.6

Description

  • Motivation
    SGWs/SGSNs in VPLMNs may reject sessions when the Create Session Response from the erGW contains unexpected values in the Bearer Level Quality of Service (Bearer QoS) in the Bearer Context IE.
  • Proposal
    Change the erGW behaviour to copy the IE received in the Create Session Request instead of returning hard coded values.

Current behavior

erGW builds this IE in the Create Session Response with static values.

Expected behavior

The Bearer QoS IE value is copied back from the Create Session Request.

Send end marker on GTP-U connection after a hand-over

When ever a handover procedure changes the GTP-U path, a end marker GPT-U frames should be send on the old path to acknowledge the successful execution.

We need to:

  1. send end marker frames to the old GTP-U peer when a remote tunnel endpoint changes
  2. tear town GTP-U tunnels when we receive a end marker that matches a running tunnel

wrong PFCP control flow during HSS Initiated Subscribed QoS Modification

HSS Initiated Subscribed QoS Modification is defined in TS 23.401 clause 5.4.2.2 and updated for CUPS in TS 23.214 clause 6.3.1.7.

The resulting control flow should invoke a PFCP session modification only after the Update Bearer Response (step in 23.401) has been received.

We currently trigger a PFCP usage query on receiving the modification command and trigger another PFCP usage query when getting the Update Bearer Response. Since QoS is currently not supported, updating the bearers in the UPF is not needed, but the first usage query is not needed at all and the second query is only needed when the QoS settings have indeed been changed.

A race between the session termination when the update bearer procedure returns a failure and the first usage query causes random test case failures in pgw_suite.modify_bearer_command_congestion.

Failure is update bearer requests are rare, so this bug has only medium priority.

Implement configurable GTP path ICMP error behaviour

erGW version

2.8.5

Description

  • Motivation
    In the network an ICMP error packet can be received on a GTP path for intermittent errors, resulting in the undesirable behaviour of immediately removing all the contexts bound to the path. The path management improvement in the linked #303 ticket is a larger change, that can take while to implement.
  • Proposal
    As a short term solution and prerequisite for ticket #303, this ticket proposes to implement a configuration option where the behaviour of the GTP path upon receiving an ICMP error can be configured.

Current behavior

ICMP error packets immediately remove all the contexts bound to the path without the possibility to influence the behaviour.

Expected behavior

erGW has a configuration option in the path_management configuration section that allows the choice of action in case of ICMP errors from the following possibilities:

  • immediate : (default) which it the current behaviour of the system
  • ignore : ignore the ICMP error and rely only on path failure detection based on echo messages

The name of the new configuration could be icmp_error_handling.

Example of new option:

{path_management, [
    {icmp_error_handling, ignore}
]}

Shuffle the config loading to make it available earlier

erGW version

2.8.5

Description

Our configuration loading and checking always had problems with when to load and apply what. This reshuffle the part to separate configuration validation and the actual application of the handler and socket config.

Current behavior

The configuration loading and checking occurs after initialization of some components.

Expected behavior

The configuration loading and checking should happen before starting and initializing components.

Add path_management to README

erGW version

2.8.5

Description

It is necessary to expand the example configuration with the addition of a missing path_management section.

Current behavior

The example of path_management section is missed.

Expected behavior

The example of path_management section is provided

APN validation

Hi

I configured ergw to only validate one APN:
{apns,
[{[<<"vservices">>], [{vrf, upstream}]}
]}

But during testing, I could use any APN and ergw would send radius authentication request to AAA servers. With correct APN and user configured on AAA, PDP context is created and logs show:
11:01:36.802 [info] AuthResult: success
11:01:36.803 [info] ActiveSessionOpts: #{'3GPP-Allocation-Retention-Priority' => 2,'3GPP-Charging-Characteristics' => <<8,0>>,'3GPP-GGSN-Address' => {10,0,0,154},'3GPP-GPRS-Negotiated-QoS-Profile' => <<11,146,31>>,'3GPP-IMSI' => <<"240010123456789">>,'3GPP-IMSI-MCC-MNC' => <<"24001">>,'3GPP-NSAPI' => 0,'3GPP-PDP-Type' => 'IPv6','3GPP-SGSN-Address' => <<10,0,0,152>>,'Acct-Authentic' => 'RADIUS','AuthProvider' => ergw_aaa_radius,'AuthProviderState' => {state,<<"ergw.conor.co.za">>,{{10,0,0,80},1812,<<"testing123">>},{{10,0,0,80},1813,<<"testing123">>},success,[],default},'Called-Station-Id' => <<"vservices">>,'Calling-Station-Id' => <<"27828099496">>,'Framed-IP-Address' => {172,25,0,1},'Framed-Protocol' => 'GPRS-PDP-Context','Interim-Accounting' => 10000,'MS-Primary-DNS-Server' => <<"\b\b\b\b">>,'MS-Primary-NBNS-Server' => <<127,0,0,1>>,'MS-Secondary-DNS-Server' => <<8,8,4,4>>,'MS-Secondary-NBNS-Server' => <<127,0,0,1>>,'Multi-Session-Id' => 66101610735641112384308940457214434475822284805,'Password' => <<"hemmelig">>,'Service-Type' => 'Framed-User','Session-Id' => 66101610735641112384308940457214434475822284806,'Username' => <<"mig">>}
11:01:36.803 [debug] handle_call: {allocate_pdp_ip,19,{172,25,0,1},undefined}
11:01:36.803 [debug] DP Call #gtp_port{name=grx,type='gtp-u',pid=<0.201.0>,restart_counter=16,ip={10,0,0,154}}: {create_pdp_context,{10,0,0,152},23,1,{172,25,0,1}}
11:01:36.803 [debug] DP Call <5226.181.0>: {create_pdp_context,{10,0,0,152},23,1,{172,25,0,1}}
11:01:36.804 [debug] DP Call Reply: {error,4026531839}
11:01:36.804 [debug] Event: Account 'Start'

With incorrect APN, AAA is still being called, but because the APN is not configured in AAA, the Auth request fails and no PDP context is created:
11:02:44.859 [info] AuthResult: fail
11:02:44.860 [error] Received DOWN information for <0.654.0> with info normal
11:02:44.860 [error] ctld Session terminating with state {state,0,<0.654.0>,#Ref<0.0.1.7738>,false,false,undefined,undefined,#{'3GPP-Allocation-Retention-Priority' => 2,'3GPP-Charging-Characteristics' => <<8,0>>,'3GPP-GGSN-Address' => {10,0,0,154},'3GPP-GPRS-Negotiated-QoS-Profile' => <<11,146,31>>,'3GPP-IMSI' => <<"240010123456789">>,'3GPP-IMSI-MCC-MNC' => <<"24001">>,'3GPP-NSAPI' => 0,'3GPP-PDP-Type' => 'IPv6','3GPP-SGSN-Address' => <<10,0,0,152>>,'AuthProvider' => ergw_aaa_radius,'AuthProviderState' => {state,<<"ergw.conor.co.za">>,{{10,0,0,80},1812,<<"testing123">>},{{10,0,0,80},1813,<<"testing123">>},fail,[],default},'Called-Station-Id' => <<"vservices1">>,'Calling-Station-Id' => <<"27828099496">>,'Framed-Protocol' => 'GPRS-PDP-Context','Interim-Accounting' => 10000,'Multi-Session-Id' => 66101610735641112384308940457214434475822284807,'Password' => <<"hemmelig">>,'Service-Type' => 'Framed-User','Session-Id' => 66101610735641112384308940457214434475822284808,'Username' => <<"mig">>}} with reason normal

I would have expected that ergw should have validated the APN and not tried to do an Auth.

Regards
Morne

Improve ICMP error handling behaviour in ERGW GTP Path maintenance

erGW version

2.8.4

Description

In specific network environments ICMP error packets can be received as a result of intermittent errors. Considering the GTP path immediately down (and closing all contexts associated with them) can be an unnecessary "harsh" action in such environments.
This feature request proposes that the GTP path on which the ICMP error was received should initiate echo probes to confirm that the error is persistent, before considering the path down.

Current behavior

Currently when an ICMP error is received for a GTP path, it immediately triggers the the path down behaviour, resulting in the teardown of all the contexts associated with that path.

Expected behavior

When the path is in up state, having active contexts bound to it, ICMP errors don't directly trigger path down, but immediately start an echo keepalive probe which then follow the retry behaviour defined by T3*N3 configuration and verify whether the path is available or not. Until the echo probe is concluded either successfully (keeping the path state) or with failure with no response after the retries (setting the path to down state), subsequent ICMP errors should be ignored.
When the path is in down state (ICMP error could be result of down echo), ICMP errors should be ignored.
When the path is in idle (no active contexts bound to it) the current behaviour (changing the path state to down) should apply.

CT; proxy_lib_SUITE is broken

commit 1f1b221 broken the suite.

It does pass when run as part of a full run, but running it standalone with rebar3 ct --suite=proxy_lib_SUITE fails because the gtp_path_reg is not started.

add `aaa` config section to schema

erGW version

master branch

Description

With the config change merged, the configuration is checked against a JSON schema. That schema does not yet contain the rules for the 'aaa' key.

Draining contexts by criteria

Feature Request

  • Create API for delete contexts by criteria:
    • Delete N contexts by criteria - /api/v1/contexts/count?apn=xyw&imsi=some_data
    • Delete all contexts by criteria - /api/v1/contexts?apn=xyw&imsi=some_data
  • For backward compatibility we should we should keep old behavior if URI is not included in the path
Name Type Description
apn ... ...
imsi ... ...
mccmnc ... ...
gtp_version ... ...

crashing during bootup when used as gtp-c-node

I am trying to run erGW in a configuration as gtp-c-node in a Docker container.
During bootup of the process inside the container, it crashed due to errors starting the netdata module.

The last lines of the log are attached for further investigation. The configuration can be found in the repo below.
There is no netdata service running inside the container.
Is this a configuration error or a faulty behavior in the code?

(the build recipe can be found at https://github.com/travelping/ergw-gtp-c-node-docker and the built docker image at https://hub.docker.com/r/grundrausch3n/ergw-gtp-c-node/)


=PROGRESS REPORT==== 15-Nov-2016::18:48:59 ===
         application: ergw_proxy_ds_json
          started_at: 'ergw-gtp-c-node@vlx162-tphh'

=SUPERVISOR REPORT==== 15-Nov-2016::18:48:59 ===
     Supervisor: {local,netdata_sup}
     Context:    start_error
     Reason:     {'EXIT',
                     {undef,
                         [{gen_statem,start_link,
                              [{local,netdata},netdata,[],[]],
                              []},
                          {supervisor,do_start_child,2,
                              [{file,"supervisor.erl"},{line,358}]},
                          {supervisor,start_children,3,
                              [{file,"supervisor.erl"},{line,341}]},
                          {supervisor,init_children,2,
                              [{file,"supervisor.erl"},{line,307}]},
                          {gen_server,init_it,6,
                              [{file,"gen_server.erl"},{line,328}]},
                          {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,240}]}]}}
     Offender:   [{pid,undefined},
                  {id,netdata},
                  {mfargs,{netdata,start_link,[]}},
                  {restart_type,permanent},
                  {shutdown,5000},
                  {child_type,worker}]


=CRASH REPORT==== 15-Nov-2016::18:48:59 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.737.0>
    registered_name: []
    exception exit: {{shutdown,
                      {failed_to_start_child,netdata,
                       {'EXIT',
                        {undef,
                         [{gen_statem,start_link,
                           [{local,netdata},netdata,[],[]],
                           []},
                          {supervisor,do_start_child,2,
                           [{file,"supervisor.erl"},{line,358}]},
                          {supervisor,start_children,3,
                           [{file,"supervisor.erl"},{line,341}]},
                          {supervisor,init_children,2,
                           [{file,"supervisor.erl"},{line,307}]},
                          {gen_server,init_it,6,
                           [{file,"gen_server.erl"},{line,328}]},
                          {proc_lib,init_p_do_apply,3,
                           [{file,"proc_lib.erl"},{line,240}]}]}}}},
                     {netdata_app,start,[normal,[]]}}
      in function  application_master:init/4 (application_master.erl, line 134)
    ancestors: [<0.736.0>]
    messages: [{'EXIT',<0.738.0>,normal}]
    links: [<0.736.0>,<0.532.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 169
  neighbours:
18:48:59.611 [error] Supervisor netdata_sup had child netdata started with netdata:start_link() at undefined exit with reason {'EXIT',{undef,[{gen_statem,start_link,[{local,netdata},netdata,[],[]],[]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,358}]},{supervisor,start_children,3,[{file,"supervisor.erl"},{line,341}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,307}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}} in context start_error
18:48:59.612 [error] CRASH REPORT Process <0.737.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,netdata,{'EXIT',{undef,[{gen_statem,start_link,[{local,netdata},netdata,[],[]],[]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,358}]},{supervisor,start_children,3,[{file,"supervisor.erl"},{line,341}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,307}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}}},{netdata_app,start,[normal,[]]}} in application_master:init/4 line 134
{"Kernel pid terminated",application_controller,"{application_start_failure,netdata,{{shutdown,{failed_to_start_child,netdata,{'EXIT',{undef,[{gen_statem,start_link,[{local,netdata},netdata,[],[]],[]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,358}]},{supervisor,start_children,3,[{file,\"supervisor.erl\"},{line,341}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,307}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}}},{netdata_app,start,[normal,[]]}}}"}
User defined signal 2

Add _total counter for path metrics

Path metrics exposed in the prometheus format are counted per peer IP (see 1). However, we also need a total counter. Usually this would be done via labels or tags. Since that would require a substantial effort: Please add a simple _total counter to the path metrics.

Example given:

 path.<SocketName>.<PeerIP>.contexts

In addition, provide path.<SocketName>.contexts.total (so it yields path_<SocketName>_contexts_total in prometheus)

Also, path.<SocketName>.<PeerIP>.rx.v1.create_pdp_context_request and friends exists. In addition, path.<SocketName>.rx.v1.create_pdp_context_requests.total etc.

Required branches: master and stable/1.x

starting failed because of missing /var/lib/ergw

I am trying to run erGW in a configuration as gtp-c-node in a Docker container.
The release was built with rebar3 and then copied.
During bootup the following error message appeared: 18:48:14.552 [error] Failed to read /var/lib/ergw/ergw.state with no such file or directory.
After creation of /var/lib/ergw/ the error did not come up again.
It would be great if the software could be patched, so that it creates the folder during bootup if it is missing.

(the build recipe can be found at https://github.com/travelping/ergw-gtp-c-node-docker and the built docker image at https://hub.docker.com/r/grundrausch3n/ergw-gtp-c-node/)

No data path rules in PFCP session establishment.

Hi, I tried setting up a data plane communication by using VPP-UPF and ERGW-PGW nodes.

The PDP context is requested by a sgsnemu node and the PGW deals with Sx Association setup and Sx Session establishment which both succed. However the PDP context response fails with error 199 (No resources available).

By an inspection of the content of the packets sniffed with wireshark it comes up that the Sx session establishment request does not provide rules to the UPF regarding the establishment of a path for the user data, which should do according to specification 29.244, but only the one for the control plane.

Is it a bug or am I missing something?

Ergw 3.0/3.1 with the included config

Hello,
The included config doesn't suit ergw 3.0/3.1 Could you adjust it ?

===> Failed to boot ergw for reason {bad_return,
                                                {{ergw_app,start,[normal,[]]},
                                                 {'EXIT',
                                                  {{badmatch,undefined},
                                                   [{ergw_config,load_config,
                                                     0,
                                                     [{file,
                                                       "/usr/local/src/ergw-3.1.1/apps/ergw/src/ergw_config.erl"},
                                                      {line,57}]},
                                                    {ergw_config,
                                                     '-load/0-fun-1-',1,
                                                     [{file,
                                                       "/usr/local/src/ergw-3.1.1/apps/ergw/src/ergw_config.erl"},
                                                      {line,52}]},
                                                    {ergw_app,start,2,
                                                     [{file,
                                                       "/usr/local/src/ergw-3.1.1/apps/ergw/src/ergw_app.erl"},
                                                      {line,24}]},
                                                    {application_master,
                                                     start_it_old,4,
                                                     [{file,
                                                       "application_master.erl"},
                                                      {line,293}]}]}}}}

Return SGW TEID in GTP session/context creation response header

erGW version

2.8.6

Description

  • Motivation
    In the network during draining, when setting the erGW not to accept new requests (reject with "no resources available"), the many SGWs/SGSNs from different networks could not correlate the session creation requests and the responses. The analysis was that the root cause of the issue is that erGW returns the GTP rejections with the TEID GTP header value set to 0. While the specs (e.g. 3GPP 129.274) leave some space for interpretation.
  • Proposal
    Implement in the returning of the SGW TEID in the session error handling code.

Current behavior

The SGW TEID is not returning in the session error handling code.

Expected behavior

The SGW TEID is returning in the session error handling code.

GTPv2 Create Session Response with "Cause: No resources available".

Hi,

I am trying to reproduce your SAE-GW walkthrough (https://github.com/travelping/ergw/blob/master/doc/walkthrough-saegw.md), however I am currently facing a problem.

When I am trying to connect a UE to the network, the MME send a GTPv2 Create Session Request and erGW answer with a GTPv2 Create Session Response with "Cause: No resources available".

I installed erGW and VPP without any problem and they are both working, PFCP messages are sent between the UP and the CP.

My configuration is the following: I use the MME and HSS from Open5GS which are working fine and are installed on my computer. The eNodeB and UE are simulated with srsLTE and are also working.
erGW and VPP are installed inside a docker container with 3 network interfaces as it is advised in the walkthrough.

I use erGW version 2.8.5 and this VPP version: https://github.com/travelping/upg-vpp/
Also, do you have an updated version of the init.conf of VPP with the UPG plugin for this walkthrough?

Thank you in advance

Fix tagging of docker images in the erGW CI builds

erGW version

2.8.4

Description

The CI builds should properly tag erGW docker images.

Current behavior

Currently the erGW build scripts do not tag docker images properly when building a tag. It tags them with branch_tag format (e.g. master_2.8.4).

Expected behavior

The docker images built by CI triggered by tagging in the repository makes a docker image available with only the tag (e.g. 2.8.4)

Delete bearer is sent instead of Delete PDP context, when deleting PDP context via HTTP API call.

erGW version

Version 3.1.1

Erlang/OTP version

Erlang/OTP 24 [erts-12.3]

Steps to reproduce

I create only one context request from SGSN, Create PDP context and delete it via API call:
DELETE http://80.240.27.68:8001/api/v1/contexts?count=1

Current behavior

Instead of Delete PDP context to SGSN (Gn interface), Delete Bearer (LTE) is sent(?!)
image

image

Also I played with gtp_context:trigger_delete_session(Context), inside Erlang VM, same thing again. I noticed during recon tracing that Interface:delete_context(... is always set to Interface=pgw_s5s8.

Expected behavior

Delete PDP context request should be sent to SGSN (Gn interface)

Config

Add metrics for successful/failed context creation

Explicit metrics for successfully created contexts/sessions as well as failed/rejected context/session creations is required -- this is an important metric for operation and cannot be reasonably deduced from the existing metrics.

These metrics should cover both situations where contexts are initiated by a peer sending requests, as well as if ergw sending requests towards a peer.

"Failed" creations could mean that the request was rejected for some reason (by ergw or peer) or no response was received for a context/session create request.

I think it makes sense to add those as per-peer metric. Ideally, the reject/fail cause is part of the metric.

Remove /metrics from Swagger definitions

erGW version

2.8.5

Description

Remove /metrics from Swagger definitions regarding discussion in PR #305.

Current behavior

The /metrics is show in Swagger.

Expected behavior

The /metrics should not show in Swagger.

UE disconnects from erGW but not from the eNodeB after a while

erGW version

2.8.11

Erlang/OTP version

23

Current behavior

Hi,
I am currently trying to connect two UEs on a network with erGW and UPG-VPP. The architecture of the configuration is below. Both of the UEs are connecting to the network and can access to the Internet, ping each other and ping other addresses on the network.

However after a while, erGW considers that the UEs are disconnected while there is no problem on the eNodeB. I can not access any other addresses nor the Internet in this state.

2021-04-26T14:19:25.268688+02:00 <0.1008.0> debug: DP Call {pfcp,v1,session_deletion_request,7820677958247942088,undefined,[]}
2021-04-26T14:19:25.268763+02:00 <0.1007.0> debug: PrepSend: {send_req,{192,168,1,1},30000,5,undefined,{pfcp,v1,session_deletion_request,7820677958247942088,undefined,[]},undefined,{ergw_sx_node,response,[<0.1008.0>,{call,{<0.1134.0>,#Ref<0.3885035724.1325137929.219164>}}]},-576460617572626536}
2021-04-26T14:19:25.268896+02:00 <0.1007.0> debug: PrepSend: Session Deletion Request(V: v1, SEID: 7820677958247942088, Seq: 34): []
2021-04-26T14:19:25.269102+02:00 <0.1007.0> debug: handle message 192.168.1.1:8805: <<33,55,0,179,108,136,161,42,212,24,135,200,0,0,34,0,0,19,0,1,1,0,79,0,158,0,81,0,4,0,0,0,1,0,104,0,4,0,0,0,0,0,63,0,3,0,8,0,0,75,0,4,228,49,45,215,0,76,0,4,228,49,46,77,0,66,0,49,63,0,0,0,0,0,14,183,174,0,0,0,0,0,0,220,26,0,0,0,0,0,13,219,148,0,0,0,0,0,0,5,4,0,0,0,0,0,0,1,163,0,0,0,0,0,0,3,97,0,67,0,4,0,0,0,118,0,69,0,4,228,49,45,216,0,70,0,4,228,49,46,76,128,3,0,10,72,249,228,49,46,77,68,222,12,0,128,4,0,10,72,249,228,49,45,215,41,226,60,0,128,5,0,10,72,249,228,49,46,77,41,226,60,0>>
2021-04-26T14:19:25.269422+02:00 <0.1007.0> debug: handle message 192.168.1.1:8805: Session Deletion Response(V: v1, SEID: 7820677958247942088, Seq: 34): [#pfcp_cause{cause = 'Request accepted'}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 118}, end_time => #end_time{time = 3828428365}, start_time => #start_time{time = 3828428247}, time_of_first_packet => #time_of_first_packet{time = 3828428248}, time_of_last_packet => #time_of_last_packet{time = 3828428364}, tp_now => #tp_now{now = -4.249350616909769e174}, tp_start_time => #tp_start_time{start = -4.248904891549682e174}, tp_stop_time => #tp_stop_time{stop = -4.249350219115811e174}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 1}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 964526,uplink = 56346,downlink = 908180, total_pkts = 1284,uplink_pkts = 419,downlink_pkts = 865}}}]
2021-04-26T14:19:25.269869+02:00 <0.1007.0> debug: <0.1007.0>: found response: 34: Session Deletion Response(V: v1, SEID: 7820677958247942088, Seq: 34): [#pfcp_cause{cause = 'Request accepted'}, #usage_report_sdr{, group =, #{duration_measurement => #duration_measurement{duration = 118}, end_time => #end_time{time = 3828428365}, start_time => #start_time{time = 3828428247}, time_of_first_packet => #time_of_first_packet{time = 3828428248}, time_of_last_packet => #time_of_last_packet{time = 3828428364}, tp_now => #tp_now{now = -4.249350616909769e174}, tp_start_time => #tp_start_time{start = -4.248904891549682e174}, tp_stop_time => #tp_stop_time{stop = -4.249350219115811e174}, ur_seqn => #ur_seqn{number = 0}, urr_id => #urr_id{id = 1}, usage_report_trigger =>, #usage_report_trigger{, immer = 0,droth = 0,stopt = 0,start = 0,quhti = 0, timth = 0,volth = 0,perio = 0,eveth = 0,macar = 0, envcl = 0,monit = 0,termr = 1,liusa = 0,timqu = 0, volqu = 0,emrre = 0,quvti = 0,ipmjl = 0,tebur = 0, evequ = 0}, volume_measurement =>, #volume_measurement{, total = 964526,uplink = 56346,downlink = 908180, total_pkts = 1284,uplink_pkts = 419,downlink_pkts = 865}}}]
2021-04-26T14:19:25.270143+02:00 <0.1007.0> debug: send_request_reply: {send_req,{192,168,1,1},30000,5,<<33,54,0,12,108,136,161,42,212,24,135,200,0,0,34,0>>,{pfcp,v1,session_deletion_request,7820677958247942088,34,[]},undefined,{ergw_sx_node,response,[<0.1008.0>,{call,{<0.1134.0>,#Ref<0.3885035724.1325137929.219164>}}]},-576460617572626536}
2021-04-26T14:19:25.270378+02:00 <0.1134.0> debug: GxSessionOpts: #{'3GPP-SGSN-MCC-MNC' => <<"20893">>,'Calling-Station-Id' => <<"03632303839333330333">>,'3GPP-MS-TimeZone' => {128,0},'Service-Type' => 'Framed-User','Framed-Protocol' => 'GPRS-PDP-Context','Called-Station-Id' => <<"Open5GS">>,'ECGI' => <<2,248,57,0,1,155,1>>,'3GPP-Selection-Mode' => 0,'TAI' => <<2,248,57,0,7>>,'3GPP-Charging-Id' => 2110551595,owner => <0.1134.0>,'MS-Primary-DNS-Server' => <<"\b\b\b\b">>,'3GPP-IMEISV' => <<"3526411107747901">>,'3GPP-RAT-Type' => 6,'Password' => <<"ergw">>,credits => #{},'MS-Secondary-DNS-Server' => <<8,8,4,4>>,'Framed-Pool' => <<"pool-A">>,'Username' => <<"ergw">>,'Node-Id' => <<"PGW-001">>,'3GPP-NSAPI' => 5,'Session-Start' => -576460735682083398,'3GPP-PDP-Type' => 'IPv4','3GPP-GGSN-MCC-MNC' => <<"20893">>,'Charging-Rule-Base-Name' => <<"rb-0001">>,'MS-Secondary-NBNS-Server' => <<127,0,0,1>>,'Session-Id' => 58493679401027239492922607821821703684,'Framed-IPv6-Pool' => <<"pool-A">>,'Diameter-Session-Id' => <<"96d79ed31a3f;738294030;481173760;647151613443">>,'Accounting-Start' => -576460735680494057,'3GPP-Charging-Characteristics' => <<84,0>>,'Framed-IP-Address' => {10,0,180,166},'3GPP-IMSI' => <<"208930110000303">>,'3GPP-User-Location-Info' => <<130,2,248,57,0,7,2,248,57,0,1,155,1>>,'3GPP-MSISDN' => <<"03632303839333330333">>,'NAS-Identifier' => <<"NAS-Identifier">>,'MS-Primary-NBNS-Server' => <<127,0,0,1>>,'QoS-Information' => #{'APN-Aggregate-Max-Bitrate-DL' => 600000,'APN-Aggregate-Max-Bitrate-UL' => 600000,'Allocation-Retention-Priority' => #{'Pre-emption-Capability' => 0,'Pre-emption-Vulnerability' => 0,'Priority-Level' => 15},'Guaranteed-Bitrate-DL' => 0,'Guaranteed-Bitrate-UL' => 0,'Max-Requested-Bandwidth-DL' => 0,'Max-Requested-Bandwidth-UL' => 0,'QoS-Class-Identifier' => 9},'Requested-IP-Address' => {10,0,180,166},'Multi-Session-Id' => 58493679401027239492922607821821703683,'Event-Trigger' => 18,'3GPP-GGSN-Address' => {172,20,16,2},now => -576460735682058609,'Bearer-Operation' => 1,'Idle-Timeout' => 28800000,'3GPP-IMSI-MCC-MNC' => <<"20893">>,'3GPP-SGSN-Address' => {192,168,56,2}}
2021-04-26T14:19:25.270981+02:00 <0.1135.0> error: Received DOWN information for <0.1134.0> with info normal

Then, when I properly disconnect the UE from the eNodeB, erGW returns this

2021-04-26T14:19:57.076948+02:00 <0.1168.0> debug: unable to find context {request,{epc,{172,20,16,1},60259,delete_session_request,{v2,20}}}
2021-04-26T14:19:57.077046+02:00 <0.1168.0> debug: unable to find context {socket_teid_key,epc,'gtp-c',1877507345}
2021-04-26T14:19:57.077119+02:00 <0.1168.0> error: handler failed with: not_found

I only need to restart erGW (not VPP) to fix this problem temporarily. The duration the UEs stay connected to erGW varies but it is generally less than a minute.

Side notes

I also noticed this two points related to the distribution of the IP addresses by erGW.

  • First, I noticed that the subnet is different between the two UEs: one of them has 10.0.180.166/30 while the other has 10.0.180.128/24
  • I had an issue once where both of the UEs had the same address attributed

Config

image
The address range for the UEs is 10.0.180.100/24 - 10.0.180.200/24

This is my erGW configuration ergw-c-node.zip

Is my problem linked to a configuration problem in erGW?
I thank you in advance

Build is broked for OTP 22.3 in ergw 2.8.7

erGW version

2.8.7

Description

Due to troubles with Travis CI it was difficult to track down the build issue for OTP 22.3.

Current behavior

Syntax error is thrown during compilation for OTP 22.3: src/ergw_sx_socket.erl:96: syntax error before: _ffff_ffff_ffff.

Expected behavior

The build for OTP 22.3 is occurs without errors.

Test cases for the ICMP behavior

erGW version

2.8.5

Description

In the current implementation are missing test cases of ICMP behavior. Need to create test cases for cover full ICMP behavior.

Current behavior

Test cases for the ICMP behavior is missed.

Expected behavior

The test cases for cover full ICMP behavior is provided

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.