Git Product home page Git Product logo

apache / trafficserver Goto Github PK

View Code? Open in Web Editor NEW
1.7K 166.0 777.0 151.62 MB

Apache Traffic Server™ is a fast, scalable and extensible HTTP/1.1 and HTTP/2 compliant caching proxy server.

Home Page: https://trafficserver.apache.org/

License: Apache License 2.0

Makefile 0.03% C++ 84.62% Perl 0.42% Shell 0.96% C 2.36% Python 9.54% Java 0.05% Lua 0.33% CMake 1.45% Dockerfile 0.05% SWIG 0.12% Objective-C 0.05% Starlark 0.01% Vim Script 0.01% Rust 0.01%
proxy cdn cache apache hacktoberfest forwardproxy http2 http3 quic reverseproxy

trafficserver's Introduction

Apache Traffic Server

Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins Jenkins

Traffic Server is a high-performance building block for cloud services. It's more than just a caching proxy server; it also has support for plugins to build large scale web applications.

Important notice to ATS developers

AS of version 10, ATS has transitioned to cmake as its build system. Below is a quick-start guide to building ATS with cmake:

Step 1: Configuration

With cmake, you definitely want to create an out-of-source build. You will give that directory to every cmake command. For these examples, it will just be build.

$ cmake -B build

This will configure the project with defaults.

If you want to customize the build, you can pass values for variables on the command line. Or, you can interactively change them using the ccmake program.

$ cmake -B build -DCMAKE_INSTALL_PREFIX=/tmp/ats -DBUILD_EXPERIMENTAL_PLUGINS=ON

-- or --

$ ccmake build

Specifying locations of dependencies

To specify the location of a dependency (like --with-* for autotools builds), you generally set a variable with the ROOT. The big exception to this is for openssl. This variable is called OPENSSL_ROOT_DIR

$ cmake -B build -Djemalloc_ROOT=/opt/jemalloc -DPCRE_ROOT=/opt/edge -DOPENSSL_ROOT_DIR=/opt/boringssl

Using presets to configure the build

cmake has a feature for grouping configurations together to make configuration and reproduction easier. The file CMakePresets.json declares presets that you can use from the command line. You can provide your own CMakeUserPresets.json and further refine those via inheritance:

$ cmake --preset dev

You can start out your user presets by just copying CMakePresets.json and removing everything in configurePresets

Here is an example user preset:


    {
      "name": "clang",
      "hidden": true,
      "environment": {
        "LDFLAGS": "-L/opt/homebrew/opt/llvm/lib -L/opt/homebrew/opt/llvm/lib/c++ -Wl,-rpath,/opt/homebrew/opt/llvm/lib/c++ -fuse-ld=/opt/homebrew/opt/llvm/bin/ld64.lld",
        "CPPFLAGS": "-I/opt/homebrew/opt/llvm/include",
        "CXXFLAGS": "-stdlib=libc++",
        "CC": "/opt/homebrew/opt/llvm/bin/clang",
        "CXX": "/opt/homebrew/opt/llvm/bin/clang++"
      }
    },
    {
      "name": "mydev",
      "displayName": "my development",
      "description": "My Development Presets",
      "binaryDir": "${sourceDir}/build-dev-clang",
      "inherits": ["clang", "dev"],
      "cacheVariables": {
        "CMAKE_INSTALL_PREFIX": "/opt/ats-cmake",
        "jemalloc_ROOT": "/opt/homebrew",
        "ENABLE_LUAJIT": false,
        "ENABLE_JEMALLOC": true,
        "ENABLE_MIMALLOC": false,
        "ENABLE_MALLOC_ALLOCATOR": true,
        "BUILD_EXPERIMENTAL_PLUGINS": true,
        "BUILD_REGRESSION_TESTING": true
      }
    },

And then use it like:

cmake --preset mydev

Building the project

$ cmake --build build
$ cmake --build build -t traffic_server

running tests

$ cd build
$ ctest

installing

$ cmake --install build

DIRECTORY STRUCTURE

trafficserver ............. Top src dir
├── ci .................... Quality assurance and other CI tools and configs
├── configs ............... Configurations
├── contrib ............... Various contributed auxiliary pieces
├── doc ................... Documentation for Traffic Server
│   ├── admin-guide ....... Admin guide documentations
│   ├── appendices ........ Appendices of Traffic Server
│   ├── developer-guide ... Documentation for developers
│   ├── dot ............... Graphviz source files for docs pictures
│   ├── static ............ Static resources
│   └── uml ............... Documentation in UML
├── example ............... Example plugins
├── iocore ................
│   ├── aio ............... Asynchronous I/O core
│   ├── cache ............. Disk and RAM cache
│   ├── dns ............... DNS (asynchronous)
│   ├── eventsystem ....... Event Driven Engine
│   ├── hostdb ............ Internal DNS cache
│   ├── net ............... Network
│   │   └── quic .......... QUIC implementation
│   └── utils ............. Utilities
├── lib ...................
│   ├── records ........... Library for config files
│   └── yamlcpp ........... Library for YAML of C++
├── m4 .................... Custom macros for configure.ac
├── mk .................... Includes for Makefiles
├── mgmt .................. JSONRPC server/management and tools
├── plugins ............... Stable core plugins
│   └── experimental ...... Experimental core plugins
├── proxy ................. HTTP proxy logic
│   ├── hdrs .............. Headers parsing and management
│   ├── http .............. The actual HTTP protocol implementation
│   ├── http2 ............. HTTP/2 implementation
│   ├── http3 ............. HTTP/3 implementation
│   ├── logging ........... Flexible logging
│   └── shared ............ Shared files
├── rc .................... Installation programs and scripts
├── src ................... Source for all the main binaries / applications
│   ├── traffic_cache_tool  Tool to interact with the Traffic Server cache
│   ├── traffic_crashlog .. Helper process that catches Traffic Server crashes
│   ├── traffic_ctl ....... Command line management tool
│   ├── traffic_layout .... Display information on the build and runtime directory structure
│   ├── traffic_logcat .... Convert binary log file to plain text
│   ├── traffic_logstats .. Log parsing and metrics calculation utility
│   ├── traffic_server .... Main proxy server
│   ├── traffic_top ....... Top like tool for viewing Traffic Server statistics
│   ├── traffic_via ....... Tool for decoding the Traffic Server Via header codes
│   ├── tscore ............ Base / core library
│   ├── tscpp ............. C++ api wrapper for plugin developers
├── tests ................. Different tests for Traffic Server
├── tools ................. Directory of various tools
├── INSTALL ............... Build and installation guide
├── LAYOUT ................ Traffic Server default layout
├── LICENSE ............... Full license text
├── NOTICE ................ Copyright notices
├── README ................ Intro, links, build info
├── README-EC2 ............ Info on EC2 support
├── REVIEWERS ............. (Incomplete) list of areas with committer interest
└── STATUS ................ Release history and information

REQUIREMENTS

This section outlines build requirements for different OS distributions. This may be out of date compared to the on-line requirements at

https://cwiki.apache.org/confluence/display/TS/Building.

As of ATS v9.0.0 and later, gcc 7 or later is required, since we now use and require the C++17 standard.

Fedora / CentOS / RHEL:

cmake
ninja
pkgconfig
gcc/g++ or clang/clang++
openssl-devel
pcre-devel
ncurses-devel and libcurl-devel(optional, needed for traffic_top)
libcap-devel (optional, highly recommended)
hwloc-devel (optional, highly recommended)

Ubuntu / Debian

cmake
ninja
pkg-config
gcc/g++ or clang/clang++
zlib1g-dev
libssl-dev
libpcre3-dev
libcap-dev (optional, highly recommended)
libhwloc-dev (optional, highly recommended)
libncurses5-dev (optional, required for e.g.: traffic_top)
libcurl4-openssl-dev (optional, required for e.g.: traffic_top)

Alpine Linux

build-base
libexecinfo-dev
pcre-dev
libressl-dev
cmake
ninja
linux-headers

macOS (we recommend HomeBrew):

cmake
ninja
pkg-config
openssl
pcre

FreeBSD

cmake
ninja
devel/gmake
devel/pkgconf
security/openssl
devel/pcre
textproc/flex (optional, install newer version from ports, fix PATH)
devel/hwloc (optional, highly recommended)

Building from distribution

You can download the latest source code from the official Apache Traffic Server site:

https://trafficserver.apache.org/downloads

(or via the URL shortener: http://s.apache.org/uG). Once downloaded, follow the instructions:

tar jxvf trafficserver-9.1.3.tar.bz2
cd trafficserver-9.1.3
cmake -B build
cmake --build build

This will build with a destination prefix of /usr/local. You can finish the installation with

sudo cmake --install build

BUILDING FROM GIT REPO

git clone https://github.com/apache/trafficserver.git   # get the source code from ASF Git repository
cd trafficserver                                        # enter the checkout directory
cmake --preset default                                  # configure the build
cmake --build build-default                             # execute the compile
cmake --build build-default -t test                     # run tests (optional)
cmake --install build-default                           # install

Instructions for building on EC2

NOTE: Alternately you may use the scripts under 'contrib' which will automate the install for trafficserver under EC2 which is HIGHLY RECOMMENDED. See 'README-EC2' for further details.

As root do the following when using Ubuntu

mkdir -p /mnt          #EC2 Storage Mount, where storage is located
cd /mnt
git clone ...          # get the source code from ASF Git repo
cd trafficserver       # enter the checkout dir
cmake --preset default                                  # configure the build
cmake --build build-default                                   # execute the compile
cmake --build build-default -t test
cmake --install build-default

As root do the following when using Fedora Core 8 kernel

mkdir -p /mnt                             #EC2 Storage Mount, where storage is located
cd /mnt
git clone ...                             # get the source code from ASF Git repo
cd trafficserver                          # enter the checkout dir
cmake --preset default                    # configure the build
cmake --build build-default               # execute the compile
cmake --build build-default -t test       # run tests (optional)
cmake --install build-default             # install

INSTALLATION

/usr/local
├── /var/log/trafficserver   log files created at runtime
├── /var/trafficserver       runtime files
├── /etc/trafficserver       configuration files
├── /bin                     executable binaries
└── /libexec/trafficserver   plugins

CRYPTO NOTICE

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software:

The functionality of OpenSSL http://www.openssl.org/ is utilized in parts of the software.

Fuzzing

FLAGS

export CC=clang
export CXX=clang++
export CFLAGS="-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link"
export CXXFLAGS="-O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fsanitize=address -fsanitize-address-use-after-scope -fsanitize=fuzzer-no-link"
export LIB_FUZZING_ENGINE=-fsanitize=fuzzer

Compile

cmake -B build -DENABLE_POSIX_CAP=OFF -DENABLE_FUZZING=ON -DYAML_BUILD_SHARED_LIBS=OFF
cmake --build build

ADDITIONAL INFO

trafficserver's People

Contributors

bgaff avatar bneradt avatar brbzull0 avatar bryancall avatar chitianhao avatar cmcfarlen avatar dragon512 avatar duke8253 avatar igalic avatar jablko avatar jacksontj avatar josiahwi avatar jpeach avatar masaori335 avatar maskit avatar mingzym avatar mlibbey avatar mturk avatar oknet avatar persiaaziz-zz avatar psudaemon avatar randall avatar shinrich avatar shukitchan avatar solidwallofcode avatar sudheerv avatar traeak avatar yunkai avatar ywkaras avatar zwoop avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

trafficserver's Issues

CID 1367527: Program hangs (SLEEP): cluster/ClusterCom.cc

*** CID 1367527:  Program hangs  (SLEEP)
/mgmt/cluster/ClusterCom.cc: 1953 in ClusterCom::sendReliableMessage(unsigned long, char *, int, char *, int, bool)()
1947           ink_mutex_release(&mutex);
1948         }
1949         close_socket(fd);
1950         return false;
1951       }
1952     
   CID 1367527:  Program hangs  (SLEEP)
   Call to "mgmt_readline" might sleep while holding lock "this->mutex".
1953       if (mgmt_readline(fd, reply, len2) == -1) {
1954         mgmt_elog(errno, "[ClusterCom::sendReliableMessage] Read failed\n");
1955         perror("ClusterCom::sendReliableMessage");
1956         reply[0] = '\0';
1957         if (take_lock) {
1958           ink_mutex_release(&mutex);

Deadlock in HostDB

We have some 7.0.0 boxes, which ends up completely wedged, where all ET_NET threads get stuck on the same lock (so, a deadlock):

#6  HostDBProcessor::getbyname_imm (this=<optimized out>, cont=cont@entry=0x2ab037b1d420, process_hostdb_info=<optimized out>, hostname=<optimized out>, len=<optimized out>, opt=...) at HostDB.cc:816
#6  HostDBProcessor::getbyname_imm (this=<optimized out>, cont=cont@entry=0x2aabc1e66a00, process_hostdb_info=<optimized out>, hostname=<optimized out>, len=<optimized out>, opt=...) at HostDB.cc:816
...

The trace is always the same in every thread:

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x00002aaaad73e5d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00002aaaad73e4a7 in __pthread_mutex_lock (mutex=0x2aaab098a290) at pthread_mutex_lock.c:61
#3  0x00002aaaaadca986 in ink_mutex_acquire (m=0x2aaab098a290) at ../../lib/ts/ink_mutex.h:90
#4  Mutex_lock (t=0x2aaab160db40, m=0x2aaab098a280) at ../../iocore/eventsystem/I_Lock.h:410
#5  MutexLock::MutexLock (t=0x2aaab160db40, am=0x2aaab098a280, this=0x2aaab470a890) at ../../iocore/eventsystem/I_Lock.h:497
#6  HostDBProcessor::getbyname_imm (this=<optimized out>, cont=cont@entry=0x2aab91432580, process_hostdb_info=<optimized out>, hostname=<optimized out>, len=<optimized out>, opt=...) at HostDB.cc:816
#7  0x00002aaaaacae21c in HttpSM::do_hostdb_lookup (this=this@entry=0x2aab91432580) at HttpSM.cc:4133
#8  0x00002aaaaacc0093 in HttpSM::set_next_state (this=0x2aab91432580) at HttpSM.cc:7248
#9  0x00002aaaaacad47a in HttpSM::call_transact_and_set_next_state (this=this@entry=0x2aab91432580, f=f@entry=0x0) at HttpSM.cc:7111
#10 0x00002aaaaacb7baf in HttpSM::handle_api_return (this=0x2aab91432580) at HttpSM.cc:1604
#11 0x00002aaaaacba5eb in HttpSM::state_api_callout (this=0x2aab91432580, event=0, data=0x0) at HttpSM.cc:1542
#12 0x00002aaaaacbf62b in HttpSM::set_next_state (this=0x2aab91432580) at HttpSM.cc:7144
#13 0x00002aaaaacad47a in HttpSM::call_transact_and_set_next_state (this=this@entry=0x2aab91432580, f=f@entry=0x0) at HttpSM.cc:7111
#14 0x00002aaaaacb9910 in HttpSM::state_hostdb_lookup (this=0x2aab91432580, event=500, data=0x2aebe3144800) at HttpSM.cc:2217
#15 0x00002aaaaacc165d in HttpSM::main_handler (this=0x2aab91432580, event=500, data=0x2aebe3144800) at HttpSM.cc:2661
#16 0x00002aaaaadc7f37 in Continuation::handleEvent (data=0x2aebe3144800, event=500, this=0x2aab91432580) at ../../iocore/eventsystem/I_Continuation.h:153
#17 reply_to_cont (cont=0x2aab91432580, r=0x2aebe3144800, is_srv=<optimized out>) at HostDB.cc:474
#18 0x00002aaaaadcc79d in HostDBContinuation::dnsEvent (this=<optimized out>, event=<optimized out>, e=<optimized out>) at HostDB.cc:1450
#19 0x00002aaaaade3821 in Continuation::handleEvent (data=<optimized out>, event=600, this=<optimized out>) at ../../iocore/eventsystem/I_Continuation.h:153
#20 DNSEntry::postEvent (this=this@entry=0x2aaab76b4e00) at DNS.cc:1269
#21 0x00002aaaaade880b in dns_result (h=h@entry=0x2aaabafc9ec0, e=e@entry=0x2aaab76b4e00, ent=<optimized out>, ent@entry=0x2aaaee3aa440, retry=retry@entry=false) at DNS.cc:1221
#22 0x00002aaaaadeb189 in dns_process (len=<optimized out>, buf=0x2aaaee3aa440, handler=0x2aaabafc9ec0) at DNS.cc:1587
#23 DNSHandler::recv_dns (this=this@entry=0x2aaabafc9ec0) at DNS.cc:782
#24 0x00002aaaaadebac9 in DNSHandler::mainEvent (this=0x2aaabafc9ec0, event=<optimized out>, e=<optimized out>) at DNS.cc:794
#25 0x00002aaaaaf0758e in Continuation::handleEvent (data=0x2aaab1788980, event=5, this=<optimized out>) at I_Continuation.h:153
#26 EThread::process_event (calling_code=5, e=0x2aaab1788980, this=0x2aaab160db40) at UnixEThread.cc:143
#27 EThread::execute (this=0x2aaab160db40) at UnixEThread.cc:270
#28 0x00002aaaaaf06136 in spawn_thread_internal (a=0x2aaab09981f0) at Thread.cc:84
#29 0x00002aaaad73caa1 in start_thread (arg=0x2aaab470c700) at pthread_create.c:301
#30 0x00002aaaae5f393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

We're not sure if this relates to HostDB sync or not, but the boxes we encountered this on, did have syncing on.

CID 1367521: Low impact quality (MISSING_MOVE_ASSIGNMENT): tsconfig/Errata.h and tsconfig/TsValue.h

*** CID 1367521:  Low impact quality  (MISSING_MOVE_ASSIGNMENT)
/lib/tsconfig/Errata.h: 82 in ()
76     
77     /** Class to hold a stack of error messages (the "errata").
78         This is a smart handle class, which wraps the actual data
79         and can therefore be treated a value type with cheap copy
80         semantics. Default construction is very cheap.
81      */
   CID 1367521:  Low impact quality  (MISSING_MOVE_ASSIGNMENT)
   Class "ts::Errata" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
82     class Errata {
83     protected:
84         /// Implementation class.
85         struct Data;
86         /// Handle for implementation class instance.
87         typedef IntrusivePtr<Data> ImpPtr;

and

*** CID 1367520:  Low impact quality  (MISSING_MOVE_ASSIGNMENT)
/lib/tsconfig/TsValue.h: 420 in ()
414     
415     /** This holds a value from the configuration.
416     
417         @internal It is critical that none of the type specific subclasses define any data members
418         so that instances can be freely converted to and from this base class.
419     */
   CID 1367520:  Low impact quality  (MISSING_MOVE_ASSIGNMENT)
   Class "ts::config::Value" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
420     class Value {
421       friend class Configuration;
422     public:
423       typedef Value self; ///< Self reference type.
424       /// Default constructors.
425       /// Creates an @c NULL instance.

CID 1368316 & 1368315: Leaks and NULL references

I think this landed recently (TS-5092?), and we have to fix this for 7.1.x. clang-analyzer also detects the NULL pointer reference.

** CID 1368316:  Resource leaks  (RESOURCE_LEAK)
/proxy/http/HttpSM.cc: 5049 in HttpSM::do_http_server_open(bool)()


________________________________________________________________________________________________________
*** CID 1368316:  Resource leaks  (RESOURCE_LEAK)
/proxy/http/HttpSM.cc: 5049 in HttpSM::do_http_server_open(bool)()
5043         int len          = 0;
5044         const char *host = t_state.hdr_info.server_request.host_get(&len);
5045         if (host && len > 0) {
5046           opt.set_sni_servername(host, len);
5047         }
5048     
   CID 1368316:  Resource leaks  (RESOURCE_LEAK)
   Failing to save or free storage allocated by "Layout::relative_to(this->t_state.txn_conf->client_cert_filepath, this->t_state.txn_conf->client_cert_filename)" leaks it.
5049         ats_scoped_str clientCert(
5050           ats_strdup((Layout::relative_to(t_state.txn_conf->client_cert_filepath, t_state.txn_conf->client_cert_filename))));
5051         opt.set_client_certname(clientCert);
5052         connect_action_handle = sslNetProcessor.connect_re(this,                                 // state machine
5053                                                            &t_state.current.server->dst_addr.sa, // addr + port
5054                                                            &opt);

** CID 1368315:  Null pointer dereferences  (FORWARD_NULL)
/proxy/http2/Http2ConnectionState.cc: 217 in rcv_headers_frame(Http2ConnectionState &, const Http2Frame &)()


________________________________________________________________________________________________________
*** CID 1368315:  Null pointer dereferences  (FORWARD_NULL)
/proxy/http2/Http2ConnectionState.cc: 217 in rcv_headers_frame(Http2ConnectionState &, const Http2Frame &)()
211         if (!stream) {
212           return error;
213         }
214       }
215     
216       // keep track of how many bytes we get in the frame
   CID 1368315:  Null pointer dereferences  (FORWARD_NULL)
   Dereferencing null pointer "stream".
217       stream->request_header_length += payload_length;
218       if (stream->request_header_length > Http2::max_request_header_size) {
219         return Http2Error(HTTP2_ERROR_CLASS_STREAM, HTTP2_ERROR_PROTOCOL_ERROR,
220                           "recv headers payload for headers greater than header length");
221       }
222     

remap plugin won't work

Brief

calling the following functions before TS_HTTP_POST_REMAP_HOOK cause rule in remap plugin don't work.

TSUrlPortSet();
TSUlPathSet();
TSUrlHostSet();
TSUrlSchemeSet();

here is one line in remap.config

map http://192.168.16.174/ http://192.168.16.174/  @plugin=libtslua.so  @pparam=/usr/local/etc/trafficserver/sta_modifyurl.lua

Detail

here is code fragment from cacheurl.cc, the left side only #if 1 and #endif is new code and the right side is a new header file to modify client request url.
the hook point is TS_HTTP_READ_REQUEST_HDR_HOOK
cacheurl

the code in above picture will cause the libtslua.so won't work, even not enter in any of the following functions in ts_lua.c (the ts-lua plugin source is here)

TSReturnCode
TSRemapInit(TSRemapInterface *api_info, char *errbuf, int errbuf_size);

TSReturnCode
TSRemapNewInstance(int argc, char *argv[], void **ih, char *errbuf, int errbuf_size);

void
TSRemapDeleteInstance(void *ih);


TSRemapStatus
TSRemapDoRemap(void *ih, TSHttpTxn rh, TSRemapRequestInfo *rri);

static int
globalHookHandler(TSCont contp, TSEvent event ATS_UNUSED, void *edata);

So, what's wrong after calling those Set functions, and how to solve this problem ?
Thank you!

CID 1367518: (LOCK): records/P_RecCore.cc

*** CID 1367518:    (LOCK)
/lib/records/P_RecCore.cc: 751 in RecSyncConfigToTB(textBuffer *, bool *)()
745                   break;
746                 default:
747                   ink_assert(!"Unexpected RecD type");
748                   break;
749                 }
750                 tb->copyFrom("\n", 1);
   CID 1367518:    (LOCK)
   "rec_mutex_release" unlocks "r->lock.the_mutex" while it is unlocked.
751                 rec_mutex_release(&(r->lock));
752               }
753             }
754             llq_rec = llq_rec->next;
755           }
756           ink_rwlock_unlock(&g_records_rwlock);
/lib/records/P_RecCore.cc: 751 in RecSyncConfigToTB(textBuffer *, bool *)()
745                   break;
746                 default:
747                   ink_assert(!"Unexpected RecD type");
748                   break;
749                 }
750                 tb->copyFrom("\n", 1);
   CID 1367518:    (LOCK)
   "rec_mutex_release" unlocks "r->lock.the_mutex" while it is unlocked.
751                 rec_mutex_release(&(r->lock));
752               }
753             }
754             llq_rec = llq_rec->next;
755           }
756           ink_rwlock_unlock(&g_records_rwlock);

CID 1367517: API usage errors (LOCK): mgmt/FileManager.cc

*** CID 1367517:  API usage errors  (LOCK)
/mgmt/FileManager.cc: 97 in FileManager::~FileManager()()
91     
92         delete rb;
93       }
94     
95       ink_hash_table_destroy(bindings);
96     
   CID 1367517:  API usage errors  (LOCK)
   "ink_mutex_destroy" destroys "this->accessLock" while it is locked.
97       ink_mutex_destroy(&accessLock);
98       ink_mutex_destroy(&cbListLock);
99     }
100     
101     // void FileManager::registerCallback(FileCallbackFunc func)
102     //

CID 1367522: (MISSING_MOVE_ASSIGNMENT): /ts/Ptr.h

*** CID 1367522:    (MISSING_MOVE_ASSIGNMENT)
/lib/ts/Ptr.h: 101 in ()
95     
96     ////////////////////////////////////////////////////////////////////////
97     //
98     // class Ptr
99     //
100     ////////////////////////////////////////////////////////////////////////
   CID 1367522:    (MISSING_MOVE_ASSIGNMENT)
   Class "Ptr<ProxyMutex>" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
101     template <class T> class Ptr
102     {
103       // https://u2389337.ct.sendgrid.net/wf/click?upn=08onrYu34A-2BWcWUl-2F-2BfV0W45MESkt7QSpvBKPCpfGg0EsS4CE-2BsucoA-2Bh5sNd9pMEAb7uIMmByan8uZ59B7ORHJPmtAdxHf-2B8vORfhUKU34-3D_3VE9LA9cRpyFj86bhmsMYTy9sYYkQM91aQkNph4gHbXkGRWVX6Y7-2BgNOjPK-2Feq5oj4AF-2FfiTCfDhIFPLYPA5maqlar3DtcXLDMevYyHoNond8zOHE6yUK6SAWNH0X9fXb-2FdeKmNgn1ZaYENzUdZPdmHX8B00b8hrDYppf7MJUzJm2OUOTa0eQEZVzQhPuxwaTpn6pR9oWSWj-2B6C-2BX0vK1w-3D-3D
104       typedef void (Ptr::*bool_type)() const;
105       void
106       this_type_does_not_support_comparisons() const
/lib/ts/Ptr.h: 101 in ()
95     
96     ////////////////////////////////////////////////////////////////////////
97     //
98     // class Ptr
99     //
100     ////////////////////////////////////////////////////////////////////////
   CID 1367522:    (MISSING_MOVE_ASSIGNMENT)
   Class "Ptr<RefCountObj>" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
101     template <class T> class Ptr
102     {
103       // https://u2389337.ct.sendgrid.net/wf/click?upn=08onrYu34A-2BWcWUl-2F-2BfV0W45MESkt7QSpvBKPCpfGg0EsS4CE-2BsucoA-2Bh5sNd9pMEAb7uIMmByan8uZ59B7ORHJPmtAdxHf-2B8vORfhUKU34-3D_3VE9LA9cRpyFj86bhmsMYTy9sYYkQM91aQkNph4gHbXkGRWVX6Y7-2BgNOjPK-2Feq5oJw3BetWxAOU-2B5T0N6CLRuD5JIZr7iaXdmIM8db3DvT7HDD4Lk9bctbunI99yTFLPPFBxy1BpxG7SILZd1aPYcwg-2F7jXuTWyNeHRZHnu6b1XjTxlfXUjJDk2XiQdeIpAcmkk6wy1Zv2djzDU1UjY1Pw-3D-3D
104       typedef void (Ptr::*bool_type)() const;
105       void
106       this_type_does_not_support_comparisons() const
/lib/ts/Ptr.h: 101 in ()
95     
96     ////////////////////////////////////////////////////////////////////////
97     //
98     // class Ptr
99     //
100     ////////////////////////////////////////////////////////////////////////
   CID 1367522:    (MISSING_MOVE_ASSIGNMENT)
   Class "Ptr<HostDBInfo>" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
101     template <class T> class Ptr
102     {
103       // https://u2389337.ct.sendgrid.net/wf/click?upn=08onrYu34A-2BWcWUl-2F-2BfV0W45MESkt7QSpvBKPCpfGg0EsS4CE-2BsucoA-2Bh5sNd9pMEAb7uIMmByan8uZ59B7ORHJPmtAdxHf-2B8vORfhUKU34-3D_3VE9LA9cRpyFj86bhmsMYTy9sYYkQM91aQkNph4gHbXkGRWVX6Y7-2BgNOjPK-2Feq5oU0SsS59AOD6tr1dderDaUQeOghEgVOzWoqqKs-2BIK-2F-2Bj795lyrUxxQqwWFVu0wzLf5eH1Fj6cjTF3YYcxBJTA6IJPhmUQhXZ3p21FswJ9FKgyW9ZmaS-2FAvi-2BtkFHvt4G3lBDuQMzT-2BBIhMJekozu2Vw-3D-3D
104       typedef void (Ptr::*bool_type)() const;
105       void
106       this_type_does_not_support_comparisons() const
/lib/ts/Ptr.h: 101 in ()
95     
96     ////////////////////////////////////////////////////////////////////////
97     //
98     // class Ptr
99     //
100     ////////////////////////////////////////////////////////////////////////
   CID 1367522:    (MISSING_MOVE_ASSIGNMENT)
   Class "Ptr<LogFile>" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
101     template <class T> class Ptr
102     {
103       // https://u2389337.ct.sendgrid.net/wf/click?upn=08onrYu34A-2BWcWUl-2F-2BfV0W45MESkt7QSpvBKPCpfGg0EsS4CE-2BsucoA-2Bh5sNd9pMEAb7uIMmByan8uZ59B7ORHJPmtAdxHf-2B8vORfhUKU34-3D_3VE9LA9cRpyFj86bhmsMYTy9sYYkQM91aQkNph4gHbXkGRWVX6Y7-2BgNOjPK-2Feq5owEAlkpEylvOeLgjNWeAM6CsvrumvSwZCQx9yoyG539fA3nT6OyyoHnhWwSmJhJ9Arw6C7dgniEN-2Fj-2BuvNmo8TjWFIitO50lMPs7w82TWD3Uac-2FJayiiIyBqRfU9sH0nSqpLtz7IHiOnKhlpH4Ozqlw-3D-3D
104       typedef void (Ptr::*bool_type)() const;
105       void
106       this_type_does_not_support_comparisons() const

Origin SNI

I currently have ATS configured to support a pristine host header.

proxy.config.url_remap.pristine_host_hdr 1

I also have ATS configured to verify the origin server certificate.

proxy.config.ssl.client.verify.server 1

My remap looks like this.

map https://edge.tld/ https://origin.tld/

Because pristine is enabled, when ATS sends a request back to the origin, it uses a SNI value of:

 edge.tld

However, the origin returns a certificate that does not match the SNI. Specifically a CN of 'origin.tld'

Because the requested SNI and the returned CN/SAN do not match, coupled with verify.server enabled, ATS throws a TLS alert and sends a 502 back to the client.

After some testing it appears that when the origin request is built, the SNI is derived from the original client HOST header. In situations where the origin certificate will not match the requested SNI value, the ATS administrator needs the ability to change the SNI accordingly.

A current work around is to use a lua script to modify the original client HOST header as the origin request is being built. This work around allows the administrator to modify the SNI to the desired value. This workaround as shown here(after cache lookup) does not impact cache key, or change the stored URL as part of an object's stored metadata. Changing the client HOST header before cache lookup impacts the cache key, etc. So an administrator must take care whenever the client HOST header is modified,.

++++++++
function cache_lookup()
ts.client_request.header['Host'] = 'origin.tld'
return 0
end
function do_remap()
ts.hook(TS_LUA_HOOK_CACHE_LOOKUP_COMPLETE, cache_lookup)
return 0
end
++++++++

Ideally there should be an over-ride option that sets the SNI which has priority over the client HOST header. One thought is to pass origin TLS options on the remap line.

Something like..

map https://edge.tld/ https://origin.tld/ tlsopt:sni=origin.tld;tlsopt2=foo;tlsopt3=bar

(maybe as a start only support a SNI tlsopt. add support for more tlsopts as ATS evolves)

Crash on retry to open server connection with POST transformed content.

We have a plugin, it is working on POST transform.

ATS crash if a client sent the request:

telnet hunter.duapp.com 80

POST /command/?command=nlog-post&channel=miller&fr=iphone&mc=iPhone%205s&av=8.2.5.10&aid=88&v=1&sv=10.1.1&s=640%2A1136&op=46003 HTTP/1.1
Host: hunter.duapp.com
Content-Type: gzip
md5: 2094631255
Accept: */* 
Cookie: BAEID=EE0DE797B09D4DF50BBD38DFDE77FD59
Connection: keep-alive
length: 152 
Accept-Language: zh-cn
Content-Length: 130 
Accept-Encoding: gzip, deflate
User-Agent: %E6%89%8B%E6%9C%BA%E7%99%BE%E5%BA%A6/8.2.5.10 CFNetwork/808.1.4 Darwin/16.1.0

0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789

The hunter.duapp.com returned :

HTTP/1.1 505 
Server: openresty
Date: Thu, 19 Jan 2017 11:18:42 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive

294
<?xml version="1.0" encoding="gb2312"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>505 - HTTP Version Not Supported</title>
<script>
    var _hmt = _hmt || [];
    (function() {
        var hm = document.createElement("script");
        hm.src = "//hm.baidu.com/hm.js?7c9ab3742865045299b00ad8031a17e3";
        var s = document.getElementsByTagName("script")[0];
        s.parentNode.insertBefore(hm, s);
    })();
</script>
</head>
<body>
<h1>505 - HTTP Version Not Supported</h1>
</body>
</html>


0

ATS will downgrade the http version and retry the request if HTTP status code is 505.

HttpTransact::handle_cache_operation_on_forward_server_response(State *s)
{
...
  case HTTP_STATUS_HTTPVER_NOT_SUPPORTED: // 505
  {
    bool keep_alive = (s->current.server->keep_alive == HTTP_KEEPALIVE);

    s->next_action = how_to_open_connection(s);
...

The next_action will be set to SM_ACTION_ORIGIN_SERVER_OPEN.
And call do_http_server_open() again, then call state_http_server_open(), handle_http_server_open()
then run into do_post_transform_open() again.
We have POST transform done already, therefore it is assert on post_transform_info.vc == NULL.

ATS will crash randomly if ignore the assert.

I think we should change the assert to :

HttpSM::do_post_transform_open()
{
  // ink_assert(post_transform_info.vc == NULL);
  if (post_transform_info.vc != NULL) {
    return NULL;
  }
...

NULL in records.config being interpreted as a string

CONFIG proxy.config.ssl.client.cert.path STRING NULL
CONFIG proxy.config.ssl.client.cert.filename STRING NULL
The two configs above make SSLClientUtils.cc: SSLInitClientContext() make a path "NULL/NULL" and try to load certificate from it which means the configs are pointing to string "NULL". Looks like a parsing error

CID 1367515, CID 1367514, CID 1367513: ts_lua plugin

*** CID 1367515:  Error handling issues  (CHECKED_RETURN)
/plugins/experimental/ts_lua/ts_lua_server_response.c: 354 in ts_lua_server_response_set_version()
348       GET_HTTP_CONTEXT(http_ctx, L);
349     
350       TS_LUA_CHECK_SERVER_RESPONSE_HDR(http_ctx);
351     
352       version = luaL_checklstring(L, 1, &len);
353     
   CID 1367515:  Error handling issues  (CHECKED_RETURN)
   Calling "sscanf" without checking return value (as is done elsewhere 61 out of 67 times).
354       sscanf(version, "%2u.%2u", &major, &minor);
355     
356       TSHttpHdrVersionSet(http_ctx->server_response_bufp, http_ctx->server_response_hdrp, TS_HTTP_VERSION(major, minor));
357     
358       return 0;

and

*** CID 1367514:  Error handling issues  (CHECKED_RETURN)
/plugins/experimental/ts_lua/ts_lua_client_request.c: 903 in ts_lua_client_request_set_version()
897       ts_lua_http_ctx *http_ctx;
898     
899       GET_HTTP_CONTEXT(http_ctx, L);
900     
901       version = luaL_checklstring(L, 1, &len);
902     
   CID 1367514:  Error handling issues  (CHECKED_RETURN)
   Calling "sscanf" without checking return value (as is done elsewhere 61 out of 67 times).
903       sscanf(version, "%2u.%2u", &major, &minor);
904     
905       TSHttpHdrVersionSet(http_ctx->client_request_bufp, http_ctx->client_request_hdrp, TS_HTTP_VERSION(major, minor));
906     
907       return 0;
908     }

and

*** CID 1367513:  Error handling issues  (CHECKED_RETURN)
/plugins/experimental/ts_lua/ts_lua_client_response.c: 370 in ts_lua_client_response_set_version()
364       GET_HTTP_CONTEXT(http_ctx, L);
365     
366       TS_LUA_CHECK_CLIENT_RESPONSE_HDR(http_ctx);
367     
368       version = luaL_checklstring(L, 1, &len);
369     
   CID 1367513:  Error handling issues  (CHECKED_RETURN)
   Calling "sscanf" without checking return value (as is done elsewhere 61 out of 67 times).
370       sscanf(version, "%2u.%2u", &major, &minor);
371     
372       TSHttpHdrVersionSet(http_ctx->client_response_bufp, http_ctx->client_response_hdrp, TS_HTTP_VERSION(major, minor));
373     
374       return 0;
375     }

Possible spurious warning in InkAPI

The InkAPI.cc code has a warning in the handle_event method:
Warning("INKCont Deletable but not deleted %d", m_event_count);

This seems to get triggered when I do not think there is an issue. This can be recreated by running the NullTransformation C++ SDK example plugin. While monitoring the diags.log file this warning can be observed sometimes while browsing. I used wget a number of times and this warning would be raised sometimes.

I added logging to the InkAPI.cc where the free() method is called and it seems that all of the continuations/vconns are being freed.

The bnull-transform example does not show this problem but it can be adapted to use the same logic as the C++ SDK and then the warning does start to appear. To do this I added a TS_HTTP_TXN_CLOSE_HOOK hook and then called TSContDestroy on the transform vconn where it was originally destroyed in the transform event handler. I have attached the source code for this.

Is my assumption correct that this warning is not required?

bnull-transform-warning.zip

traffic_server deadlocked after config reload

There was a configuration change(an update to parent.config) pushed to a set of servers. Shortly after the config reload(via traffic_ctl), one host's traffic_cop starting failing heartbeats. The ATS process quit serving traffic. Attaching gdb, I see a number of threads attempting to do a hostdb lookup. Filesystem has a host.db.syncing file that is dated soon after the reload.

At the time of reload, there were approximately 800 active server connections. The same configuration was applied to 23 other hosts at the same time, which successfully reloaded without issue.

syslog:

Jan  3 16:29:37 s_sys@host traffic_manager[7156]: {0x7f6ec8ffe700} NOTE: User has changed config file parent.config
Jan  3 16:29:45 s_sys@host traffic_server[7169]: {0x2aaab470c700} NOTE: loading SSL certificate configuration from /opt/user/etc/trafficserver/ssl_multicert.config
Jan  3 16:34:50 s_sys@host traffic_cop[7154]: (test) read timeout [180000 ]
Jan  3 16:34:50 s_sys@host traffic_cop[7154]: server heartbeat failed [1]
Jan  3 16:38:00 s_sys@host traffic_cop[7154]: (test) read timeout [180000 ]
Jan  3 16:38:00 s_sys@host traffic_cop[7154]: server heartbeat failed [2]

/var/cache/trafficserver:

[user@host trafficserver]$ ls -altr
total 28
drwxr-xr-x. 10 root    root     4096 Oct 17 09:00 ..
-rw-r--r--   1 user user 12029 Jan  3 16:31 host.db
drwxr-xr-x   2 user user  4096 Jan  3 16:31 .
-rw-r--r--   1 user user  4109 Jan  3 16:32 host.db.syncing

stack.txt

CID 1367531: Uninitialized members (UNINIT_CTOR): proxy/main.cc

*** CID 1367531:  Uninitialized members  (UNINIT_CTOR)
/proxy/Main.cc: 352 in MemoryLimit::MemoryLimit()()
346       }
347     };
348     
349     class MemoryLimit : public Continuation
350     {
351     public:
   CID 1367531:  Uninitialized members  (UNINIT_CTOR)
   Non-static class member field "_usage.ru_stime" is not initialized in this constructor nor in any functions that it calls.
352       MemoryLimit() : Continuation(new_ProxyMutex()), _memory_limit(0) { SET_HANDLER(&MemoryLimit::periodic); }
353       ~MemoryLimit() { mutex = NULL; }
354       int
355       periodic(int event, Event *e)
356       {
357         if (event == EVENT_IMMEDIATE) {

[openbsd] build fails while building SSLUtils.cc

Build fails on OpenBSD.

kern.version=OpenBSD 6.0-current (GENERIC.MP) #0: Wed Nov 23 20:54:10 MST 2016
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
c++ -DHAVE_CONFIG_H -I. -I../../lib  -I../../iocore/eventsystem -I../../iocore/net -I../../iocore/aio -I../../iocore/hostdb -I../../iocore/cache -I../../iocore/cluster -I../../iocore/utils -I../../iocore/dns -I../../lib -I../../lib/records -I../../proxy -I../../proxy/hdrs -I../../proxy/shared -I../../proxy/logging -I../../mgmt -I../../mgmt/utils -I../../proxy/api/ts -I../../proxy/http -I/usr/include -I/usr/local/include-Dopenbsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -D_FORTIFY_SOURCE=2 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.5  -O2 -pipe -O0 -g -std=c++11 -Wall -Wextra -Wno-ignored-qualifiers -Wno-unused-parameter -feliminate-unused-debug-symbols -fno-strict-aliasing -Wno-invalid-offsetof -fPIE -fstack-protector -mcx16 -MT SSLUtils.o -MD -MP -MF $depbase.Tpo -c -o SSLUtils.o SSLUtils.cc &&\
mv -f $depbase.Tpo $depbase.Po
SSLUtils.cc: In function 'void SSLInitializeLibrary()':
SSLUtils.cc:917:83: error: invalid conversion from 'void* (*)(size_t, const char*, int) {aka void* (*)(long unsigned int, const char*, int)}' to 'void* (*)(size_t) {aka void* (*)(long unsigned int)}' [-fpermissive]
       CRYPTO_set_mem_functions(ssl_track_malloc, ssl_track_realloc, ssl_track_free);
                                                                                   ^
In file included from /usr/include/openssl/bio.h:69:0,
                 from /usr/include/openssl/evp.h:67,
                 from /usr/include/openssl/hmac.h:67,
                 from /usr/include/openssl/ssl.h:149,
                 from P_SSLNetVConnection.h:41,
                 from P_Net.h:104,
                 from SSLUtils.cc:26:
/usr/include/openssl/crypto.h:412:5: note: initializing argument 1 of 'int CRYPTO_set_mem_functions(void* (*)(size_t), void* (*)(void*, size_t), void (*)(void*))'
 int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *, size_t), void (*f)(void *));
     ^
SSLUtils.cc:917:83: error: invalid conversion from 'void* (*)(void*, size_t, const char*, int) {aka void* (*)(void*, long unsigned int, const char*, int)}' to 'void* (*)(void*, size_t) {aka void* (*)(void*, longunsigned int)}' [-fpermissive]
       CRYPTO_set_mem_functions(ssl_track_malloc, ssl_track_realloc, ssl_track_free);
                                                                                   ^
In file included from /usr/include/openssl/bio.h:69:0,
                 from /usr/include/openssl/evp.h:67,
                 from /usr/include/openssl/hmac.h:67,
                 from /usr/include/openssl/ssl.h:149,
                 from P_SSLNetVConnection.h:41,
                 from P_Net.h:104,
                 from SSLUtils.cc:26:
/usr/include/openssl/crypto.h:412:5: note: initializing argument 2 of 'int CRYPTO_set_mem_functions(void* (*)(size_t), void* (*)(void*, size_t), void (*)(void*))'
 int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *, size_t), void (*f)(void *));
     ^
SSLUtils.cc:917:83: error: invalid conversion from 'void (*)(void*, const char*, int)' to 'void (*)(void*)' [-fpermissive]
       CRYPTO_set_mem_functions(ssl_track_malloc, ssl_track_realloc, ssl_track_free);
                                                                                   ^
In file included from /usr/include/openssl/bio.h:69:0,
                 from /usr/include/openssl/evp.h:67,
                 from /usr/include/openssl/hmac.h:67,
                 from /usr/include/openssl/ssl.h:149,
                 from P_SSLNetVConnection.h:41,
                 from P_Net.h:104,
                 from SSLUtils.cc:26:
/usr/include/openssl/crypto.h:412:5: note: initializing argument 3 of 'int CRYPTO_set_mem_functions(void* (*)(size_t), void* (*)(void*, size_t), void (*)(void*))'
 int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *, size_t), void (*f)(void *));
     ^
SSLUtils.cc:919:65: error: invalid conversion from 'void* (*)(size_t, const char*, int) {aka void* (*)(long unsigned int, const char*, int)}' to 'void* (*)(size_t) {aka void* (*)(long unsigned int)}' [-fpermissive]
       CRYPTO_set_mem_functions(ssl_malloc, ssl_realloc, ssl_free);
                                                                 ^
In file included from /usr/include/openssl/bio.h:69:0,
                 from /usr/include/openssl/evp.h:67,
                 from /usr/include/openssl/hmac.h:67,
                 from /usr/include/openssl/ssl.h:149,
                 from P_SSLNetVConnection.h:41,
                 from P_Net.h:104,
                 from SSLUtils.cc:26:
/usr/include/openssl/crypto.h:412:5: note: initializing argument 1 of 'int CRYPTO_set_mem_functions(void* (*)(size_t), void* (*)(void*, size_t), void (*)(void*))'
 int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *, size_t), void (*f)(void *));
     ^
SSLUtils.cc:919:65: error: invalid conversion from 'void* (*)(void*, size_t, const char*, int) {aka void* (*)(void*, long unsigned int, const char*, int)}' to 'void* (*)(void*, size_t) {aka void* (*)(void*, longunsigned int)}' [-fpermissive]
       CRYPTO_set_mem_functions(ssl_malloc, ssl_realloc, ssl_free);
                                                                 ^
In file included from /usr/include/openssl/bio.h:69:0,
                 from /usr/include/openssl/evp.h:67,
                 from /usr/include/openssl/hmac.h:67,
                 from /usr/include/openssl/ssl.h:149,
                 from P_SSLNetVConnection.h:41,
                 from P_Net.h:104,
                 from SSLUtils.cc:26:
/usr/include/openssl/crypto.h:412:5: note: initializing argument 2 of 'int CRYPTO_set_mem_functions(void* (*)(size_t), void* (*)(void*, size_t), void (*)(void*))'
 int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *, size_t), void (*f)(void *));
     ^
SSLUtils.cc:919:65: error: invalid conversion from 'void (*)(void*, const char*, int)' to 'void (*)(void*)' [-fpermissive]
       CRYPTO_set_mem_functions(ssl_malloc, ssl_realloc, ssl_free);
                                                                 ^
In file included from /usr/include/openssl/bio.h:69:0,
                 from /usr/include/openssl/evp.h:67,
                 from /usr/include/openssl/hmac.h:67,
                 from /usr/include/openssl/ssl.h:149,
                 from P_SSLNetVConnection.h:41,
                 from P_Net.h:104,
                 from SSLUtils.cc:26:
/usr/include/openssl/crypto.h:412:5: note: initializing argument 3 of 'int CRYPTO_set_mem_functions(void* (*)(size_t), void* (*)(void*, size_t), void (*)(void*))'
 int CRYPTO_set_mem_functions(void *(*m)(size_t), void *(*r)(void *, size_t), void (*f)(void *));
     ^
SSLUtils.cc:941:34: error: 'SSL_pthreads_thread_id' was not declared in this scope
     CRYPTO_THREADID_set_callback(SSL_pthreads_thread_id);
                                  ^
SSLUtils.cc: In function 'SSL_CTX* SSLInitServerContext(const SSLConfigParams*, const ssl_user_config*, Vec<x509_st*>&)':
SSLUtils.cc:1585:56: error: invalid conversion from 'SSL_SESSION* (*)(SSL*, const unsigned char*, int, int*) {aka ssl_session_st* (*)(ssl_st*, const unsigned char*, int, int*)}' to 'SSL_SESSION* (*)(ssl_st*, unsigned char*, int, int*) {aka ssl_session_st* (*)(ssl_st*, unsigned char*, int, int*)}' [-fpermissive]
     SSL_CTX_sess_set_get_cb(ctx, ssl_get_cached_session);
                                                        ^
In file included from P_SSLNetVConnection.h:41:0,
                 from P_Net.h:104,
                 from SSLUtils.cc:26:
/usr/include/openssl/ssl.h:932:6: note: initializing argument 2 of 'void SSL_CTX_sess_set_get_cb(SSL_CTX*, SSL_SESSION* (*)(ssl_st*, unsigned char*, int, int*))'
 void SSL_CTX_sess_set_get_cb(SSL_CTX *ctx,
      ^

[openbsd] Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`

Hi,

trafficserver built from commitid 87ca6b4 (with #1313, #1317, #1328 diffs) doesn't run correctly on OpenBSD -current amd64.

traffic_cop doesn't start subdaemons (I can't use your RC script as there's no support and OpenBSD 3rd party rc scripts work little bit different anyway).

# su -l -c daemon -s /bin/sh _trafficserver -c "/usr/bin/ktrace -di -f /tmp/kdump /usr/local/bin/traffic_cop -o -d"
No home directory /nonexistent!
Logging in with home = "/".
<1485102848.0000> [DEBUG]: Entering init()
<1485102848.0000> [DEBUG]: Entering init_signals()
<1485102848.0000> [DEBUG]: Entering set_alarm_death()
<1485102848.0000> [DEBUG]: Leaving set_alarm_death()
<1485102848.0000> [DEBUG]: Leaving init_signals()
<1485102848.0000> [DEBUG]: Entering init_config_file()
<1485102848.0000> [DEBUG]: Leaving init_config_file()
<1485102848.0000> [DEBUG]: Entering config_reload_records()
<1485102848.0000> [DEBUG]: binpath is bin
<1485102848.0000> [DEBUG]: Entering process_syslog_config()
<1485102848.0000> [DEBUG]: Leaving process_syslog_config()
<1485102848.0000> [DEBUG]: Leaving config_reload_records()
<1485102848.0000> [DEBUG]: Entering init_lockfiles()
<1485102848.0000> [DEBUG]: Leaving init_lockfiles()
<1485102848.0000> [DEBUG]: Entering check_lockfile()
<1485102848.0000> [NOTICE]: --- Cop Starting [Version: Apache Traffic Server - traffic_cop - 7.1.0 - (build # 012216 on Jan 22 2017 at 16:16:37)] ---
<1485102848.0000> [DEBUG]: Leaving check_lockfile()
<1485102848.0000> [DEBUG]: Leaving init()
<1485102848.0000> [DEBUG]: Entering check()
<1485102848.0000> [DEBUG]: Entering check_no_run()
<1485102848.0000> [DEBUG]: Entering transient_error(2, 500)
<1485102848.0000> [DEBUG]: Leaving transient_error(2, 500) --> false
<1485102848.0000> [DEBUG]: Leaving check_no_run() --> 0
<1485102848.0000> [DEBUG]: Entering config_reload_records()
<1485102848.0000> [DEBUG]: Entering check_programs()
<1485102848.0000> [WARNING]: traffic_manager not running, making sure traffic_server is dead
<1485102848.0000> [DEBUG]: Entering safe_kill(/var/run/trafficserver/server.lock, traffic_server, 0)
<1485102848.0000> [DEBUG]: Entering set_alarm_warn()
<1485102848.0000> [DEBUG]: Leaving set_alarm_warn()
<1485102848.0000> [DEBUG]: Entering set_alarm_death()
<1485102848.0000> [DEBUG]: Leaving set_alarm_death()
<1485102848.0000> [DEBUG]: Leaving safe_kill(/var/run/trafficserver/server.lock, traffic_server, 0)
<1485102848.0000> [DEBUG]: Entering milliseconds()
<1485102848.0000> [DEBUG]: Leaving milliseconds()
<1485102848.0000> [WARNING]: spawning traffic_manager
<1485102848.0000> [DEBUG]: binpath is bin
<1485102848.0000> [DEBUG]: Entering spawn_manager()
<1485102848.0000> [DEBUG]: launching /usr/local/bin/traffic_manager'
<1485102848.0000> [DEBUG]: Leaving spawn_manager()
<1485102848.0000> [DEBUG]: Leaving check_programs()
<1485102848.0000> [DEBUG]: Entering check_memory()
<1485102848.0000> [DEBUG]: Leaving check_memory()
<1485102848.0000> [DEBUG]: Entering millisleep(10000)
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
<1485102848.0000> [DEBUG]: Entering sig_child(20)
<1485102848.0000> [DEBUG]: Leaving sig_child(20)
<1485102848.0000> [DEBUG]: Leaving millisleep(10000)
<1485102848.0000> [DEBUG]: performing additional sleep for 0 sec during init<1485102848.0000> [DEBUG]: Entering millisleep(0)
<1485102848.0000> [DEBUG]: Leaving millisleep(0)
<1485102848.0000> [DEBUG]: Entering check_no_run()
<1485102848.0000> [DEBUG]: Entering transient_error(2, 500)
<1485102848.0000> [DEBUG]: Leaving transient_error(2, 500) --> false
<1485102848.0000> [DEBUG]: Leaving check_no_run() --> 0
<1485102848.0000> [WARNING]: cop received non-normal child status signal [64249 0]
<1485102848.0000> [WARNING]: child terminated due to signal 6: Abort trap
<1485102848.0000> [DEBUG]: Entering config_reload_records()
<1485102848.0000> [DEBUG]: Entering check_programs()
<1485102848.0000> [WARNING]: traffic_manager not running, making sure traffic_server is dead
...
# cat /var/log/trafficserver/traffic.out
Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`
Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`
Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`
Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`
Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`
Fatal: ink_sys_control.cc:59: failed assertion `setrlimit(MAGIC_CAST(which), &rl) >= 0`
# traffic_cop -V
Apache Traffic Server - traffic_cop - 7.1.0 - (build # 012216 on Jan 22 2017 at 16:16:37)#

Full ktrace https://gist.github.com/jirib/a50596131759e9cb79cf5a905d77db5d

[openbsd] BaseLogFile.cc:244:61: error: invalid conversion from 'long int*' to 'time_t* {aka long long int*}' [-fpermissive]

Build fails on BaseLogFile.cc.

$ sysctl kern.version
kern.version=OpenBSD 6.0-current (GENERIC.MP) #117: Sat Jan  7 09:10:45 MST 2017
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Making all in ts
gmake[3]: Entering directory '/home/jirib/openbsd/pobj/trafficserver-20170106/trafficserver-24347df553e91660b794af1079fbf69f986dc5b4/lib/ts'
depbase=`echo BaseLogFile.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/usr/bin/libtool  --tag=CXX   --mode=compile c++ -DHAVE_CONFIG_H -I. -I../../lib  -I/home/jirib/openbsd/pobj/trafficserver-20170106/trafficserver-24347df553e91660b794af1079fbf69f986dc5b4/lib -I/usr/local/include
-Dopenbsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -D_FORTIFY_SOURCE=2 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.5  -O2 -pipe -O0 -g -std=c++11 -Wall -Wextra -Wno-ignored-qualifiers -Wno-unused-parameter -feliminate-unused-debug-symbols -fno-strict-aliasing -Wno-invalid-offsetof -fPIE -fstack-protector -mcx16 -MT BaseLogFile.lo -MD -MP -MF $depbase.Tpo -c -o BaseLogFile.lo BaseLogFile.cc &&\
mv -f $depbase.Tpo $depbase.Plo
c++ -DHAVE_CONFIG_H -I. -I../../lib -I/home/jirib/openbsd/pobj/trafficserver-20170106/trafficserver-24347df553e91660b794af1079fbf69f986dc5b4/lib -I/usr/local/include -Dopenbsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -D_FORTIFY_SOURCE=2 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.5 -O2 -pipe -O0 -g -std=c++11 -Wall -Wextra -Wno-ignored-qualifiers -Wno-unused-parameter -feliminate-unused-debug-symbols -fno-strict-aliasing -Wno-invalid-offsetof -fstack-protector -mcx16 -MT BaseLogFile.lo -MD -MP -MF .deps/BaseLogFile.Tpo -c BaseLogFile.cc -fPIC -DPIC -o .libs/BaseLogFile.o
BaseLogFile.cc: In member function 'int BaseLogFile::roll()':
BaseLogFile.cc:244:61: error: invalid conversion from 'long int*' to 'time_t* {aka long long int*}' [-fpermissive]
   if (!m_meta_info || !m_meta_info->get_creation_time(&start))
                                                             ^
In file included from BaseLogFile.cc:24:0:
BaseLogFile.h:126:3: note: initializing argument 1 of 'bool BaseMetaInfo::get_creation_time(time_t*)'
   get_creation_time(time_t *time)
   ^
Error while executing c++ -DHAVE_CONFIG_H -I. -I../../lib -I/home/jirib/openbsd/pobj/trafficserver-20170106/trafficserver-24347df553e91660b794af1079fbf69f986dc5b4/lib -I/usr/local/include -Dopenbsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -D_FORTIFY_SOURCE=2 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.5 -O2 -pipe -O0
-g -std=c++11 -Wall -Wextra -Wno-ignored-qualifiers -Wno-unused-parameter -feliminate-unused-debug-symbols -fno-strict-aliasing -Wno-invalid-offsetof -fstack-protector -mcx16 -MT BaseLogFile.lo -MD -MP -MF .deps/BaseLogFile.Tpo -c BaseLogFile.cc -fPIC -DPIC -o .libs/BaseLogFile.o

CID 1368306 & 1368305: NULLPTR and security BP in logcat.cc

** CID 1368306:  Security best practices violations  (TOCTOU)
/proxy/logcat.cc: 299 in main()


________________________________________________________________________________________________________
*** CID 1368306:  Security best practices violations  (TOCTOU)
/proxy/logcat.cc: 299 in main()
293     
294       if (n_file_arguments) {
295         int bin_ext_len   = strlen(LOG_FILE_BINARY_OBJECT_FILENAME_EXTENSION);
296         int ascii_ext_len = strlen(LOG_FILE_ASCII_OBJECT_FILENAME_EXTENSION);
297     
298         for (unsigned i = 0; i < n_file_arguments; ++i) {
   CID 1368306:  Security best practices violations  (TOCTOU)
   Calling function "open" that uses "file_arguments[i]" after a check function. This can cause a time-of-check, time-of-use race condition.
299           int in_fd = open(file_arguments[i], O_RDONLY);
300           if (in_fd < 0) {
301             fprintf(stderr, "Error opening input file %s: ", file_arguments[i]);
302             perror(0);
303             error = DATA_PROCESSING_ERROR;
304           } else {

** CID 1368305:  Null pointer dereferences  (REVERSE_INULL)
/plugins/experimental/money_trace/money_trace.cc: 129 in mt_check_request_header(tsapi_httptxn *)()


________________________________________________________________________________________________________
*** CID 1368305:  Null pointer dereferences  (REVERSE_INULL)
/plugins/experimental/money_trace/money_trace.cc: 129 in mt_check_request_header(tsapi_httptxn *)()
123             txn_data                                   = allocTransactionData();
124             txn_data->client_request_mt_header         = TSstrndup(hdr_value, length);
125             txn_data->client_request_mt_header[length] = '\0'; // workaround for bug in core.
126             LOG_DEBUG("found money trace header: %s, length: %d", txn_data->client_request_mt_header, length);
127             if (nullptr == (contp = TSContCreate(transaction_handler, nullptr))) {
128               LOG_ERROR("failed to create the transaction handler continuation");
   CID 1368305:  Null pointer dereferences  (REVERSE_INULL)
   Null-checking "txn_data" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
129               if (nullptr != txn_data) {
130                 TSfree(txn_data->client_request_mt_header);
131                 TSfree(txn_data);
132               }
133             } else {
134               TSContDataSet(contp, txn_data);

CID 1367523: (MISSING_MOVE_ASSIGNMENT): tsconfig/IntrusivePtr.

*** CID 1367523:    (MISSING_MOVE_ASSIGNMENT)
/lib/tsconfig/IntrusivePtr.h: 134 in ()
128         ownded by a set of pointers. When the last of the pointers is destructed
129         the target object is also destructed.
130     
131         The smart pointer actions can be changed through class specific policy
132         by specializing the @c IntrusivePtrPolicy template class.
133     */
   CID 1367523:    (MISSING_MOVE_ASSIGNMENT)
   Class "ts::IntrusivePtr<ts::Errata::Data>" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
134     template <typename T> class IntrusivePtr : private IntrusivePtrBase
135     {
136     private:                          /* don't pollute client with these typedefs */
137       typedef IntrusivePtrBase super; ///< Parent type.
138       typedef IntrusivePtr self;      ///< Self reference type.
139     
/lib/tsconfig/IntrusivePtr.h: 134 in ()
128         ownded by a set of pointers. When the last of the pointers is destructed
129         the target object is also destructed.
130     
131         The smart pointer actions can be changed through class specific policy
132         by specializing the @c IntrusivePtrPolicy template class.
133     */
   CID 1367523:    (MISSING_MOVE_ASSIGNMENT)
   Class "ts::IntrusivePtr<ts::config::detail::ValueTableImpl>" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
134     template <typename T> class IntrusivePtr : private IntrusivePtrBase
135     {
136     private:                          /* don't pollute client with these typedefs */
137       typedef IntrusivePtrBase super; ///< Parent type.
138       typedef IntrusivePtr self;      ///< Self reference type.
139     

CID 1196420: Error handling issues (CHECKED_RETURN): proxy/CoreUtils.cc

*** CID 1196420:  Error handling issues  (CHECKED_RETURN)
/proxy/CoreUtils.cc: 482 in CoreUtils::print_http_hdr(HTTPHdr *, const char *)()
476       if (h->m_heap && h->m_http) {
477         int r = load_http_hdr(h, &new_handle);
478     
479         if (r > 0) {
480           printf("----------- %s  ------------\n", name);
481           new_handle.m_mime = new_handle.m_http->m_fields_impl;
   CID 1196420:  Error handling issues  (CHECKED_RETURN)
   Calling "print" without checking return value (as is done elsewhere 26 out of 28 times).
482           new_handle.print(nullptr, 0, nullptr, nullptr);
483           printf("-----------------------------\n\n");
484         }
485       }
486     }
487     

Possibly crasher in HostDB serialization (disk sync)

Hi Thomas,

we're seeing this odd looking crash, fairly often:

#0  0x00002aaaad74337d in __libc_waitpid (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>)
    at ../sysdeps/unix/sysv/linux/waitpid.c:41
#1  0x00002aaaaab83297 in crash_logger_invoke (signo=11, info=0x2aaaf20007f0, ctx=0x2aaaf20006c0) at Crash.cc:164
#2  <signal handler called>
#3  0x0000000000000000 in ?? ()
#4  0x00002aaaaaf06270 in Continuation::handleEvent (data=0x2ae608f1b140, event=1, this=<optimized out>) at I_Continuation.h:153
#5  EThread::process_event (this=0x2aaaf1c0da00, e=0x2ae608f1b140, calling_code=1) at UnixEThread.cc:143
#6  0x00002aaaaaf0727f in EThread::execute (this=0x2aaaf1c0da00) at UnixEThread.cc:197
#7  0x00002aaaaaf05ba6 in spawn_thread_internal (a=0x2aaac51fccc0) at Thread.cc:84
#8  0x00002aaaad73baa1 in start_thread (arg=0x2aaaf2001700) at pthread_create.c:301
#9  0x00002aaaae5f293d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Not particularly helpful, I know, but in frame #4, Randal found that the data seems related to the serialization continuation:

(gdb) frame 4
#4  0x00002aaaaaf06270 in Continuation::handleEvent (data=0x2ae608f1b140, event=1, this=<optimized out>) at I_Continuation.h:153
153	I_Continuation.h: No such file or directory.
(gdb) print *((Event*)data).continuation
$1 = {<force_VFPT_to_top> = {_vptr.force_VFPT_to_top = 0x2aaaab1d9670 <vtable for RefCountCacheSerializer<RefCountCache<HostDBInfo> >+16>},
  handler = (int (Continuation::*)(Continuation * const, int,
    void *)) 0x2aaaaadcf7a0 <RefCountCacheSerializer<RefCountCache<HostDBInfo> >::initialize_storage(int, Event*)>, mutex = {
    m_ptr = 0x2aaaf0e46e40}, link = {<SLink<Continuation>> = {next = 0x0}, prev = 0x0}, control_flags = {raw_flags = 0}}

We're thinking of turning off HostDB sync again to see if that helps, but let me know if you have any brilliant ideas as to why this would fail here (unfortunately the above is all we got :-/).

new warning running 7.1

On startup of 7.1.0, I now see this warning:

WARNING: connect_attempts_rr_retries (3) is greater than connect_attempts_max_retries (3), this means requests will never redispatch to another server

These are the values records.config.defaults to. Perhaps the default needs to change or the log warning logic needs to change.

This warning was added in bc84981

ATS 6.1.1 -- Found some orphaned connections after sending 403-response to the client.

We have ATS 6.1.1 (running under Linux RH72) configured as a reverse proxy. If the client requests a URL that is not defined with a rule in remap.config, then we send a 403 forbidden back to the client and close the connection. This normally works, but we found after several weeks (and thousands of 403's sent) that there are a handful of connections that show up as ESTABLISHED in netstat, but they do not appear to be managed any longer by ATS.

First, we monitored using tcpdump to confirm that there was no activity on these connections for about an hour. Second, we removed the cache from production and dynamically turned on debug for the net_queue tag using traffic_ctl, and even when the queue sizes were zero the established connections persisted. So these seem to be orphaned connections not policed by inactivity_cop (time-out set for 300 seconds). We confirmed in our ATS log files that ATS had previously sent completed 403 responses on these connections (matched by client IP and source port) from days up to weeks back.

Any ideas? Also, does or can ATS make use of the socket TCP level time-outs (setsockopt ... SO_KEEPALIVE)?

action=never-cache did NOT take effect in cache.config

version: trafficserver-7.0.0-rc2
testing URL: curl -v http://test.com:65432/data/1.php

first try:
################### cache.config #####################
dest_domain=test.com suffix=php action=never-cache cache-responses-to-cookies=0 #1
dest_domain=test.com suffix=jsp action=never-cache cache-responses-to-cookies=0 #2
dest_domain=test.com suffix=asp action=never-cache cache-responses-to-cookies=0 #3
dest_domain=test.com suffix=aspx action=never-cache cache-responses-to-cookies=0 #4
dest_domain=test.com suffix=ts ttl-in-cache=3s cache-responses-to-cookies=1 #5
dest_domain=test.com suffix=m3u8 ttl-in-cache=2s cache-responses-to-cookies=1 #6
dest_domain=test.com ttl-in-cache=4s cache-responses-to-cookies=0 #7
################### traffic.out #####################
[Jan 17 11:45:44.038] Server {0x2b309a30d700} DEBUG: <CacheControl.cc:466 (UpdateMatch)> (cache_control) Matched with for NEVER_CACHE at line 1 [cache-responses-to-cookies=0]
[Jan 17 11:45:44.038] Server {0x2b309a30d700} DEBUG: <CacheControl.cc:466 (UpdateMatch)> (cache_control) Matched with for TTL_IN_CACHE at line 7 [cache-responses-to-cookies=0]
##################### problem #######################
I expected rule #1, but rule #7 took effect.

second try:
################### cache.config #####################
dest_domain=test.com ttl-in-cache=4s cache-responses-to-cookies=0 #1
dest_domain=test.com suffix=php action=never-cache cache-responses-to-cookies=0 #2
dest_domain=test.com suffix=jsp action=never-cache cache-responses-to-cookies=0 #3
dest_domain=test.com suffix=asp action=never-cache cache-responses-to-cookies=0 #4
dest_domain=test.com suffix=aspx action=never-cache cache-responses-to-cookies=0 #5
dest_domain=test.com suffix=ts ttl-in-cache=3s cache-responses-to-cookies=1 #6
dest_domain=test.com suffix=m3u8 ttl-in-cache=2s cache-responses-to-cookies=1 #7
################### traffic.out #####################
[Jan 17 12:01:44.553] Server {0x2b521db00700} DEBUG: <CacheControl.cc:466 (UpdateMatch)> (cache_control) Matched with for TTL_IN_CACHE at line 1 [cache-responses-to-cookies=0]
##################### problem #######################
I expected rule #2, but rule #1 took effect.

cache promote plugin takes too much calculating size of LRUList

pstack shows that cache promote plugin spends lot of time calculating too much time in determining the size of LRUList

#0 std::__distance<std::_List_const_iterator<std::pair<LRUHash, unsigned int> > > (__last=..., __first=...) at /opt/gcc/include/c++/4.9.3/bits/stl_iterator_base_funcs.h:83
#1 std::distance<std::_List_const_iterator<std::pair<LRUHash, unsigned int> > > (__last=..., __first=...) at /opt/gcc/include/c++/4.9.3/bits/stl_iterator_base_funcs.h:118
#2 std::list<std::pair<LRUHash, unsigned int>, std::allocator<std::pair<LRUHash, unsigned int> > >::size (this=0x2b37fe095c58) at /opt/gcc/include/c++/4.9.3/bits/stl_list.h:887

using a length variable should make it not dependent on stl list implementation

Core dump in SSLAccept / SSLNetVConnection::sslServerHandShakeEvent

We (@randall) ran into an odd crashes that seems fairly similar (but not identical) to https://issues.apache.org/jira/browse/TS-4363. Since Jira is now read-only, I'm opening this up here as a new Issue (but, odds are that this is the same as the TS-4363 issue).

Backtrace:

#0  0x00002aaaad74437d in __libc_waitpid (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>)
    at ../sysdeps/unix/sysv/linux/waitpid.c:41
#1  0x00002aaaaab83297 in crash_logger_invoke (signo=11, info=0x2aaab610a8f0, ctx=0x2aaab610a7c0) at Crash.cc:164
#2  <signal handler called>
#3  0x00002aaaac715005 in ?? () from /opt/bazinga/lib/libcrypto.so.10
#4  0x00002aaaac7155cc in lh_retrieve () from /opt/bazinga/lib/libcrypto.so.10
#5  0x00002aaaac672c6f in ?? () from /opt/bazinga/lib/libcrypto.so.10
#6  0x00002aaaac67369b in ?? () from /opt/bazinga/lib/libcrypto.so.10
#7  0x00002aaaac700247 in DH_new_method () from /opt/bazinga/lib/libcrypto.so.10
#8  0x00002aaaac70170e in DHparams_dup () from /opt/bazinga/lib/libcrypto.so.10
#9  0x00002aaaac3d717a in ssl_cert_dup () from /opt/bazinga/lib/libssl.so.10
#10 0x00002aaaac3d56a4 in SSL_set_SSL_CTX () from /opt/bazinga/lib/libssl.so.10
#11 0x00002aaaaaea3724 in set_context_cert (ssl=ssl@entry=0x2aea13f9f800) at SSLUtils.cc:361
#12 0x00002aaaaaea3ad6 in ssl_cert_callback (ssl=0x2aea13f9f800) at SSLUtils.cc:398
#13 0x00002aaaac3abd40 in ssl3_get_client_hello () from /opt/bazinga/lib/libssl.so.10
#14 0x00002aaaac3b0f3f in ssl3_accept () from /opt/bazinga/lib/libssl.so.10
#15 0x00002aaaac3c0230 in ssl23_accept () from /opt/bazinga/lib/libssl.so.10
#16 0x00002aaaaaeaa383 in SSLAccept (ssl=0x2aea13f9f800) at SSLUtils.cc:2265
#17 0x00002aaaaae97a4f in SSLNetVConnection::sslServerHandShakeEvent (this=this@entry=0x2aea3b4c23e0, err=@0x2aaab610b980: 0)
    at SSLNetVConnection.cc:1068
#18 0x00002aaaaae9a058 in SSLNetVConnection::sslServerHandShakeEvent (this=0x2aea3b4c23e0, err=<optimized out>) at SSLNetVConnection.cc:1014
#19 0x00002aaaaae9a4c9 in SSLNetVConnection::sslStartHandShake (this=0x2aea3b4c23e0, event=<optimized out>, err=@0x2aaab610b980: 0)
    at SSLNetVConnection.cc:979
#20 0x00002aaaaae9cfc3 in SSLNetVConnection::net_read_io (this=0x2aea3b4c23e0, nh=0x2aaab30116f0, lthread=0x2aaab300d9c0) at SSLNetVConnection.cc:451
#21 0x00002aaaaaeaf254 in NetHandler::mainNetEvent (this=0x2aaab30116f0, event=<optimized out>, e=<optimized out>) at UnixNet.cc:526
#22 0x00002aaaaaf0758e in Continuation::handleEvent (data=0x2aaab17893a0, event=5, this=<optimized out>) at I_Continuation.h:153
#23 EThread::process_event (calling_code=5, e=0x2aaab17893a0, this=0x2aaab300d9c0) at UnixEThread.cc:143
#24 EThread::execute (this=0x2aaab300d9c0) at UnixEThread.cc:270
#25 0x00002aaaaaf06136 in spawn_thread_internal (a=0x2aaab0998460) at Thread.cc:84
#26 0x00002aaaad73caa1 in start_thread (arg=0x2aaab610c700) at pthread_create.c:301
#27 0x00002aaaae5f393d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

I'm curious if @oknet recognizes this as well? This is running v7.0.0, with a small number of cherry picks:

* 0412dec (HEAD -> 7.0.x, origin/7.0.x) TS-5102 parent metrics
* cd53220 Fix for: TS-5108-SSL requests might stall because enabled write is ignored
* d908dc0 TS-5069: Fixes CID 1366771 and 1366771
* c95d0ff TS-5076: Test and set the in_enabled_list by atomic function
* 486cc02 TS-4429: Fixes missing curly brackets with new option
* eff5595 TS-4429: Adds a --concise (-C) option for logstats, excludes % metrics

CID 1367528: Security best practices violations (STRING_OVERFLOW): http/HttpTransactHeaders.cc

*** CID 1367528:  Security best practices violations  (STRING_OVERFLOW)
/proxy/http/HttpTransactHeaders.cc: 820 in HttpTransactHeaders::insert_via_header_in_response(HttpTransact::State *, HTTPHdr *)()
814     
815       char *incoming_via = s->via_string;
816     
817       const char *proto_buf[10]; // 10 seems like a reasonable number of protos to print
818       int retval = s->state_machine->populate_client_protocol(proto_buf, countof(proto_buf));
819       for (int i = 0; i < retval; i++) {
   CID 1367528:  Security best practices violations  (STRING_OVERFLOW)
   You might overrun the 1024-character fixed-size string "via_string" by copying "proto_buf[i]" without checking the length.
820         memcpy(via_string, proto_buf[i], strlen(proto_buf[i]));
821         via_string += strlen(proto_buf[i]);
822         *via_string++ = ' ';
823       }
824     
825       via_string += nstrcpy(via_string, s->http_config_param->proxy_hostname);

CID 1367519: Low impact quality (MISSING_MOVE_ASSIGNMENT): ts/ink_memory.h

*** CID 1367519:  Low impact quality  (MISSING_MOVE_ASSIGNMENT)
/lib/ts/ink_memory.h: 242 in ()
236         assignment operator. This will be easier with C++ eleventy.
237     
238     */
239     
240     template <typename TRAITS ///< Traits object.
241               >
   CID 1367519:  Low impact quality  (MISSING_MOVE_ASSIGNMENT)
   Class "ats_scoped_resource<detail::SCOPED_MALLOC_TRAITS<char> >" may benefit from adding a move assignment operator. See other events which show the copy assignment operator being applied to rvalue(s), where a move assignment may be faster.
242     class ats_scoped_resource
243     {
244     public:
245       typedef TRAITS Traits;                          ///< Make template arg available.
246       typedef typename TRAITS::value_type value_type; ///< Import value type.
247       typedef ats_scoped_resource self;               ///< Self reference type.

CID 1367526: CID 1367525 : money_trace/money_trace.cc

*** CID 1367526:  Null pointer dereferences  (REVERSE_INULL)
/plugins/experimental/money_trace/money_trace.cc: 57 in freeTransactionData(txndata *)()
51         TSfree(txn_data->client_request_mt_header);
52       }
53       if (txn_data->new_span_mt_header != NULL) {
54         LOG_DEBUG("freeing txn_data->new_span_mt_header.");
55         TSfree(txn_data->new_span_mt_header);
56       }
   CID 1367526:  Null pointer dereferences  (REVERSE_INULL)
   Null-checking "txn_data" suggests that it may be null, but it has already been dereferenced on all paths leading to the check.
57       if (txn_data != NULL) {
58         LOG_DEBUG("freeing txn_data.");
59         TSfree(txn_data);
60       }
61     }
62     

and

*** CID 1367525:  Resource leaks  (RESOURCE_LEAK)
/plugins/experimental/money_trace/money_trace.cc: 147 in mt_check_request_header(tsapi_httptxn *)()
141           LOG_DEBUG("no money trace header was found in the request.");
142         }
143       } else {
144         LOG_DEBUG("failed to retrieve the client request.");
145       }
146       TSHandleMLocRelease(bufp, hdr_loc, field_loc);
   CID 1367525:  Resource leaks  (RESOURCE_LEAK)
   Variable "txn_data" going out of scope leaks the storage it points to.
147     }
148     
149     /**
150      * The TS_EVENT_HTTP_SEND_RESPONSE_HDR callback.
151      *
152      * Adds the money trace header received in the client request to the

Can trafficserver proxy to another proxy?

So, I have a third party proxy (probably under squid) which will only accept connections from one of my IP's, but I need to be able to access it from a variety of IPs.

So I'm trying to put a trafficserver to forward requests to this proxy. I tried the following:

regex_map http://(.*)/ http://myproxy:80/

But I get in logs status 502 (Server Hangup) for my requests, I guess because trafficserver is proxying like this:

http_proxy=http://trafficserver.com curl -v http://www.targetdomain.com.br/index.html

> GET /index.html HTTP/1.1
> Host: www.targetdomain.com.br

But I need it to proxy like this:

http_proxy=http://trafficserver.com curl -v http://www.targetdomain.com.br/index.html

> GET http://www.targetdomain.com.br/index.html HTTP/1.1

So that the third party proxy understands it.

Can trafficserver do this?

[openbsd] pthread_np.h needs to be included

This passes at least some first steps in trying to build trafficserver on OpenBSD. Please consider this or better variation, thank you.

kern.version=OpenBSD 6.0-current (GENERIC.MP) #0: Wed Nov 23 20:54:10 MST 2016
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
--- lib/ts/ink_thread.h.orig    Thu Oct 20 21:08:55 2016
+++ lib/ts/ink_thread.h Sat Jan  7 21:16:56 2017
@@ -44,6 +44,10 @@
 #include <signal.h>
 #include <semaphore.h>

+#if defined(openbsd)
+#include <pthread_np.h>
+#endif
+
 #if HAVE_PTHREAD_NP_H
 #include <pthread_np.h>
 #endif

Slowloris vulnerability

Traffic server is "likely vulnerable" to slowloris (see https://nmap.org/nsedoc/scripts/http-slowloris-check.html) in both reverse and forward proxy mode.

Has anyone investigated this yet? Is there any plan to do anything about it? I appreciate the nature of the vulnerability means fully fixing it is likely impossible, I'm just trying to find out if there's an official position on the issue.

Thanks.

openssl errors for every request

With ATS 7.1, the server will come up cleanly (same configs that work fine under 7.0.0), but a request will generate the following log lines.

Jan 18 17:36:05 hostname traffic_server[380]: {0x2ae6a690c700} ERROR: SSL::47170125350656:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE Jan 18 17:36:05 hostname traffic_server[380]: {0x2ae6a690c700} ERROR: SSL::47170125350656:error:140DC009:SSL routines:SSL_CTX_use_certificate_chain_file:PEM lib:ssl_rsa.c:708 Jan 18 17:36:05 hostname traffic_server[380]: {0x2ae6a690c700} ERROR: failed to load client certificate from (null)

The request completes successfully, but is very noisy server side.

[openbsd] build fails as openbsd doesn't have gethostbyname_r

OpenBSD doesn't have gethostbyname_r, see https://github.com/openbsd/src/blob/master/include/netdb.h

c++ -DHAVE_CONFIG_H -I. -I../../lib -I/home/jirib/openbsd/pobj/trafficserver-20170106/trafficserver-24347df553e91660b794af1079fbf69f986dc5b4/lib -I/usr/local/include -I/usr/local/include/lua-5.1/ -Dopenbsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -D_FORTIFY_SOURCE=2 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.5 -O2 -pipe -O0 -g -std=c++11 -Wall -Wextra -Wno-ignored-qualifiers -Wno-unused-parameter -feliminate-unused-debug-symbols -fno-strict-aliasing -Wno-invalid-offsetof -fstack-protector -mcx16 -MT ink_inet.lo -MD -MP -MF .deps/ink_inet.Tpo -c ink_inet.cc -fPIC -DPIC -o .libs/ink_inet.o
ink_inet.cc: In function 'hostent* ink_gethostbyname_r(char*, ink_gethostbyname_r_data*)':
ink_inet.cc:62:116: error: 'gethostbyname_r' was not declared in this scope
   struct hostent *r = gethostbyname_r(hostname, &data->ent, data->buf, INK_GETHOSTBYNAME_R_DATA_SIZE, &data->herrno);
                                                                                                                    ^
ink_inet.cc: In function 'hostent* ink_gethostbyaddr_r(char*, int, int, ink_gethostbyaddr_r_data*)':
ink_inet.cc:82:129: error: 'gethostbyaddr_r' was not declared in this scope
   struct hostent *r = gethostbyaddr_r((char *)ip, len, type, &data->ent, data->buf, INK_GETHOSTBYNAME_R_DATA_SIZE, &data->herrno);
                                                                                                                                 ^
Error while executing c++ -DHAVE_CONFIG_H -I. -I../../lib -I/home/jirib/openbsd/pobj/trafficserver-20170106/trafficserver-24347df553e91660b794af1079fbf69f986dc5b4/lib -I/usr/local/include -I/usr/local/include/lua-5.1/ -Dopenbsd -D_LARGEFILE64_SOURCE=1 -D_COMPILE64BIT_SOURCE=1 -D_GNU_SOURCE -D_REENTRANT -D__STDC_LIMIT_MACROS=1 -D__STDC_FORMAT_MACROS=1 -D_FORTIFY_SOURCE=2 -I/usr/include -DOPENSSL_NO_SSL_INTERN -I/usr/local/include/tcl8.5 -O2 -pipe -O0 -g -std=c++11 -Wall -Wextra -Wno-ignored-qualifiers -Wno-unused-parameter -feliminate-unused-debug-symbols -fno-strict-aliasing -Wno-invalid-offsetof -fstack-protector -mcx16 -MT ink_inet.lo -MD -MP -MF .deps/ink_inet.Tpo -c ink_inet.cc -fPIC -DPIC -o .libs/ink_inet.o

CID 1367529: Security best practices violations (STRING_OVERFLOW): url_sig/url_sig.c

*** CID 1367529:  Security best practices violations  (STRING_OVERFLOW)
/plugins/experimental/url_sig/url_sig.c: 489 in TSRemapDoRemap()
483       /* find the string that was signed - cycle through the parts letters, adding the part of the fqdn/path if it is 1 */
484       p = strstr(url, "?");
485       memcpy(urltokstr, &url[strlen("https://u2389337.ct.sendgrid.net/wf/click?upn=EC-2B-2FWtUtRiE7N-2BTLQqyBaw-3D-3D_3VE9LA9cRpyFj86bhmsMYTy9sYYkQM91aQkNph4gHbXkGRWVX6Y7-2BgNOjPK-2Feq5ojKy4hXb8AilyEnfeoFM8FVTDCUBzQbOSOuN0SaYI-2F963hVGPq1-2F5BsZhITVSD9CsFxXXPxHI7NuJU7MRfDpdE2EmH0fKXahkVO8OHbCTKDJ-2Ffpx9eDVKto4-2BxEjTgxJCjx-2F5p50SDRObQzmBNMrlOQ-3D-3D")], p - url - strlen("https://u2389337.ct.sendgrid.net/wf/click?upn=EC-2B-2FWtUtRiE7N-2BTLQqyBaw-3D-3D_3VE9LA9cRpyFj86bhmsMYTy9sYYkQM91aQkNph4gHbXkGRWVX6Y7-2BgNOjPK-2Feq5oqQqJlMRu9ldolQgnwmwiK7sUp7uj-2Bab0dt37qT332AaByZEMwp7v5i0rX4GwAfagcKvrCcj7ylZjfFCDT03-2Fom6onuJD4Stg1h8VlCrfAE-2F5JSmHW8Gwj8i8HMxhGFtoo2yYz1QIX4loLvr1YM0epg-3D-3D"));
486       part = strtok_r(urltokstr, "/", &p);
487       while (part != NULL) {
488         if (parts[j] == '1') {
   CID 1367529:  Security best practices violations  (STRING_OVERFLOW)
   You might overrun the 8192-character fixed-size string "&signed_part[strlen(signed_part)]" by copying "part" without checking the length.
489           strcpy(signed_part + strlen(signed_part), part);
490           strcpy(signed_part + strlen(signed_part), "/");
491         }
492         if (parts[j + 1] == '0' ||
493             parts[j + 1] == '1') { // This remembers the last part, meaning, if there are no more valid letters in parts
494           j++;                     // will keep repeating the value of the last one

relative_to(char* path, char* filename) should return null if path is null

Digging down the code, ink_filepath_merge(char *path, int pathsz, const char *rootpath, const char *addpath, int flags) concatenates the current working directory with the filename if the rootpath is null. If this is expected to do this, then at least, we should not let relative_to function call ink_filepath_merge if either filepath or filename is null

also set_paths_helper should check if either filename or filepath is null before calling the concat function

[openbsd] wrong mandir for OpenBSD layout

$ sed -n '/<Layout OpenBSD>/,/<\/Layout>/p' trafficserver-87ca6b40999b13562aff0efce5a1cd76eab5c306/config.layout  | grep man
    mandir:        ${prefix}/share/man

But 3rd party apps on OpenBSD put their man pages into ${prefix}/man.

CID 1367516: Null pointer dereferences (FORWARD_NULL): wccp/WccpConfig.cc

*** CID 1367516:  Null pointer dereferences  (FORWARD_NULL)
/lib/wccp/WccpConfig.cc: 528 in wccp::CacheImpl::loadServicesFromFile(const char *)()
522         zret.pull(load_routers(prop, Seed_Router).doNotLog());
523       }
524     
525       int idx, nsvc;
526       for (idx = 0, nsvc = svc_list.childCount(); idx < nsvc; ++idx) {
527         int x; // scratch int.
   CID 1367516:  Null pointer dereferences  (FORWARD_NULL)
   Assigning: "md5_key" = "NULL".
528         const char *md5_key = 0;
529         ts::ConstBuffer text;
530         SecurityOption security_style = SECURITY_NONE;
531         bool use_group_local_security = false;
532         Value const &svc_cfg          = svc_list[idx];
533         int svc_line                  = svc_cfg.getSourceLine();

socks.config - dest_ip rule issue

I have added a rule to the socks.config file to use a SOCKS server for a specific destination IP with the dest_ip primary specifier but it does not appear to be working. I am able to change my system setting to use the SOCKS server and that does work, I just can't get ATS to use it via the rule in socks.config.

Any help would be much appreciated!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.