Git Product home page Git Product logo

apple / swift-nio Goto Github PK

View Code? Open in Web Editor NEW
7.8K 190.0 621.0 8.52 MB

Event-driven network application framework for high performance protocol servers & clients, non-blocking.

Home Page: https://swiftpackageindex.com/apple/swift-nio/documentation

License: Apache License 2.0

Shell 1.77% Swift 91.64% C 6.36% Ruby 0.07% DTrace 0.03% Dockerfile 0.01% Python 0.11%
swift asynchronous-io networking event-driven high-performance non-blocking-io non-blocking swiftnio swift5 swift-server

swift-nio's Introduction

sswg:graduated|104x20

SwiftNIO

SwiftNIO is a cross-platform asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients.

It's like Netty, but written for Swift.

Repository organization

The SwiftNIO project is split across multiple repositories:

Repository NIO 2 (Swift 5.7+)
https://github.com/apple/swift-nio
SwiftNIO core
from: "2.0.0"
https://github.com/apple/swift-nio-ssl
TLS (SSL) support
from: "2.0.0"
https://github.com/apple/swift-nio-http2
HTTP/2 support
from: "1.0.0"
https://github.com/apple/swift-nio-extras
useful additions around SwiftNIO
from: "1.0.0"
https://github.com/apple/swift-nio-transport-services
first-class support for macOS, iOS, tvOS, and watchOS
from: "1.0.0"
https://github.com/apple/swift-nio-ssh
SSH support
.upToNextMinor(from: "0.2.0")

NIO 2.29.0 and older support Swift 5.0+, NIO 2.39.0 and older support Swift 5.2+.

Within this repository we have a number of products that provide different functionality. This package contains the following products:

  • NIO. This is an umbrella module exporting NIOCore, NIOEmbedded and NIOPosix.
  • NIOCore. This provides the core abstractions and types for using SwiftNIO (see "Conceptual Overview" for more details). Most NIO extension projects that provide things like new EventLoops and Channels or new protocol implementations should only need to depend on NIOCore.
  • NIOPosix. This provides the primary [EventLoopGroup], EventLoop, and Channels for use on POSIX-based systems. This is our high performance core I/O layer. In general, this should only be imported by projects that plan to do some actual I/O, such as high-level protocol implementations or applications.
  • NIOEmbedded. This provides EmbeddedChannel and EmbeddedEventLoop, implementations of the NIOCore abstractions that provide fine-grained control over their execution. These are most often used for testing, but can also be used to drive protocol implementations in a way that is decoupled from networking altogether.
  • NIOConcurrencyHelpers. This provides a few low-level concurrency primitives that are used by NIO implementations, such as locks and atomics.
  • NIOFoundationCompat. This extends a number of NIO types for better interoperation with Foundation data types. If you are working with Foundation data types such as Data, you should import this.
  • NIOTLS. This provides a few common abstraction types for working with multiple TLS implementations. Note that this doesn't provide TLS itself: please investigate swift-nio-ssl and swift-nio-transport-services for concrete implementations.
  • NIOHTTP1. This provides a low-level HTTP/1.1 protocol implementation.
  • NIOWebSocket. This provides a low-level WebSocket protocol implementation.
  • NIOTestUtils. This provides a number of helpers for testing projects that use SwiftNIO.
  • NIOFileSystem. This provides async APIs for interacting with the file system.

Protocol Implementations

Below you can find a list of a few protocol implementations that are done with SwiftNIO. This is a non-exhaustive list of protocols that are either part of the SwiftNIO project or are accepted into the SSWG's incubation process. All of the libraries listed below do all of their I/O in a non-blocking fashion using SwiftNIO.

Low-level protocol implementations

Low-level protocol implementations are often a collection of ChannelHandlers that implement a protocol but still require the user to have a good understanding of SwiftNIO. Often, low-level protocol implementations will then be wrapped in high-level libraries with a nicer, more user-friendly API.

Protocol Client
(Sends requests)
Server
(Responds to requests)
Repository Module Comment
HTTP/1 apple/swift-nio NIOHTTP1 official NIO project
HTTP/2 apple/swift-nio-http2 NIOHTTP2 official NIO project
WebSocket apple/swift-nio NIOWebSocket official NIO project
TLS apple/swift-nio-ssl NIOSSL official NIO project
SSH apple/swift-nio-ssh NIOSSH official NIO project

High-level implementations

High-level implementations are usually libraries that come with an API that doesn't expose SwiftNIO's ChannelPipeline and can therefore be used with very little (or no) SwiftNIO-specific knowledge. The implementations listed below do still do all of their I/O in SwiftNIO and integrate really well with the SwiftNIO ecosystem.

Protocol Client
(Sends requests)
Server
(Responds to requests)
Repository Module Comment
HTTP swift-server/async-http-client AsyncHTTPClient SSWG community project
gRPC grpc/grpc-swift GRPC also offers a low-level API; SSWG community project
APNS swift-server-community/APNSwift APNSwift SSWG community project
PostgreSQL vapor/postgres-nio PostgresNIO SSWG community project
Redis swift-server/RediStack RediStack SSWG community project

Supported Versions

SwiftNIO 2

This is the current version of SwiftNIO and will be supported for the foreseeable future.

The most recent versions of SwiftNIO support Swift 5.7 and newer. The minimum Swift version supported by SwiftNIO releases are detailed below:

SwiftNIO Minimum Swift Version
2.0.0 ..< 2.30.0 5.0
2.30.0 ..< 2.40.0 5.2
2.40.0 ..< 2.43.0 5.4
2.43.0 ..< 2.51.0 5.5.2
2.51.0 ..< 2.60.0 5.6
2.60.0 ..< 2.65.0 5.7
2.65.0 ... 5.8

SwiftNIO 1

SwiftNIO 1 is considered end of life - it is strongly recommended that you move to a newer version. The Core NIO team does not actively work on this version. No new features will be added to this version but PRs which fix bugs or security vulnerabilities will be accepted until the end of May 2022.

If you have a SwiftNIO 1 application or library that you would like to migrate to SwiftNIO 2, please check out the migration guide we prepared for you.

The latest released SwiftNIO 1 version supports Swift 4.0, 4.1, 4.2, and 5.0.

Supported Platforms

SwiftNIO aims to support all of the platforms where Swift is supported. Currently, it is developed and tested on macOS and Linux, and is known to support the following operating system versions:

Compatibility

SwiftNIO follows SemVer 2.0.0 with a separate document declaring SwiftNIO's Public API.

What this means for you is that you should depend on SwiftNIO with a version range that covers everything from the minimum SwiftNIO version you require up to the next major version. In SwiftPM that can be easily done specifying for example from: "2.0.0" meaning that you support SwiftNIO in every version starting from 2.0.0 up to (excluding) 3.0.0. SemVer and SwiftNIO's Public API guarantees should result in a working program without having to worry about testing every single version for compatibility.

Conceptual Overview

SwiftNIO is fundamentally a low-level tool for building high-performance networking applications in Swift. It particularly targets those use-cases where using a "thread-per-connection" model of concurrency is inefficient or untenable. This is a common limitation when building servers that use a large number of relatively low-utilization connections, such as HTTP servers.

To achieve its goals SwiftNIO extensively uses "non-blocking I/O": hence the name! Non-blocking I/O differs from the more common blocking I/O model because the application does not wait for data to be sent to or received from the network: instead, SwiftNIO asks for the kernel to notify it when I/O operations can be performed without waiting.

SwiftNIO does not aim to provide high-level solutions like, for example, web frameworks do. Instead, SwiftNIO is focused on providing the low-level building blocks for these higher-level applications. When it comes to building a web application, most users will not want to use SwiftNIO directly: instead, they'll want to use one of the many great web frameworks available in the Swift ecosystem. Those web frameworks, however, may choose to use SwiftNIO under the covers to provide their networking support.

The following sections will describe the low-level tools that SwiftNIO provides, and provide a quick overview of how to work with them. If you feel comfortable with these concepts, then you can skip right ahead to the other sections of this README.

Basic Architecture

The basic building blocks of SwiftNIO are the following 8 types of objects:

All SwiftNIO applications are ultimately constructed of these various components.

EventLoops and EventLoopGroups

The basic I/O primitive of SwiftNIO is the event loop. The event loop is an object that waits for events (usually I/O related events, such as "data received") to happen and then fires some kind of callback when they do. In almost all SwiftNIO applications there will be relatively few event loops: usually only one or two per CPU core the application wants to use. Generally speaking event loops run for the entire lifetime of your application, spinning in an endless loop dispatching events.

Event loops are gathered together into event loop groups. These groups provide a mechanism to distribute work around the event loops. For example, when listening for inbound connections the listening socket will be registered on one event loop. However, we don't want all connections that are accepted on that listening socket to be registered with the same event loop, as that would potentially overload one event loop while leaving the others empty. For that reason, the event loop group provides the ability to spread load across multiple event loops.

In SwiftNIO today there is one EventLoopGroup implementation, and two EventLoop implementations. For production applications there is the MultiThreadedEventLoopGroup, an EventLoopGroup that creates a number of threads (using the POSIX pthreads library) and places one SelectableEventLoop on each one. The SelectableEventLoop is an event loop that uses a selector (either kqueue or epoll depending on the target system) to manage I/O events from file descriptors and to dispatch work. These EventLoops and EventLoopGroups are provided by the NIOPosix module. Additionally, there is the EmbeddedEventLoop, which is a dummy event loop that is used primarily for testing purposes, provided by the NIOEmbedded module.

EventLoops have a number of important properties. Most vitally, they are the way all work gets done in SwiftNIO applications. In order to ensure thread-safety, any work that wants to be done on almost any of the other objects in SwiftNIO must be dispatched via an EventLoop. EventLoop objects own almost all the other objects in a SwiftNIO application, and understanding their execution model is critical for building high-performance SwiftNIO applications.

Channels, Channel Handlers, Channel Pipelines, and Channel Contexts

While EventLoops are critical to the way SwiftNIO works, most users will not interact with them substantially beyond asking them to create EventLoopPromises and to schedule work. The parts of a SwiftNIO application most users will spend the most time interacting with are Channels and ChannelHandlers.

Almost every file descriptor that a user interacts with in a SwiftNIO program is associated with a single Channel. The Channel owns this file descriptor, and is responsible for managing its lifetime. It is also responsible for processing inbound and outbound events on that file descriptor: whenever the event loop has an event that corresponds to a file descriptor, it will notify the Channel that owns that file descriptor.

Channels by themselves, however, are not useful. After all, it is a rare application that doesn't want to do anything with the data it sends or receives on a socket! So the other important part of the Channel is the ChannelPipeline.

A ChannelPipeline is a sequence of objects, called ChannelHandlers, that process events on a Channel. The ChannelHandlers process these events one after another, in order, mutating and transforming events as they go. This can be thought of as a data processing pipeline; hence the name ChannelPipeline.

All ChannelHandlers are either Inbound or Outbound handlers, or both. Inbound handlers process "inbound" events: events like reading data from a socket, reading socket close, or other kinds of events initiated by remote peers. Outbound handlers process "outbound" events, such as writes, connection attempts, and local socket closes.

Each handler processes the events in order. For example, read events are passed from the front of the pipeline to the back, one handler at a time, while write events are passed from the back of the pipeline to the front. Each handler may, at any time, generate either inbound or outbound events that will be sent to the next handler in whichever direction is appropriate. This allows handlers to split up reads, coalesce writes, delay connection attempts, and generally perform arbitrary transformations of events.

In general, ChannelHandlers are designed to be highly re-usable components. This means they tend to be designed to be as small as possible, performing one specific data transformation. This allows handlers to be composed together in novel and flexible ways, which helps with code reuse and encapsulation.

ChannelHandlers are able to keep track of where they are in a ChannelPipeline by using a ChannelHandlerContext. These objects contain references to the previous and next channel handler in the pipeline, ensuring that it is always possible for a ChannelHandler to emit events while it remains in a pipeline.

SwiftNIO ships with many ChannelHandlers built in that provide useful functionality, such as HTTP parsing. In addition, high-performance applications will want to provide as much of their logic as possible in ChannelHandlers, as it helps avoid problems with context switching.

Additionally, SwiftNIO ships with a few Channel implementations. In particular, it ships with ServerSocketChannel, a Channel for sockets that accept inbound connections; SocketChannel, a Channel for TCP connections; and DatagramChannel, a Channel for UDP sockets. All of these are provided by the NIOPosix module. It also provides EmbeddedChannel, a Channel primarily used for testing, provided by the NIOEmbedded module.

A Note on Blocking

One of the important notes about ChannelPipelines is that they are thread-safe. This is very important for writing SwiftNIO applications, as it allows you to write much simpler ChannelHandlers in the knowledge that they will not require synchronization.

However, this is achieved by dispatching all code on the ChannelPipeline on the same thread as the EventLoop. This means that, as a general rule, ChannelHandlers must not call blocking code without dispatching it to a background thread. If a ChannelHandler blocks for any reason, all Channels attached to the parent EventLoop will be unable to progress until the blocking call completes.

This is a common concern while writing SwiftNIO applications. If it is useful to write code in a blocking style, it is highly recommended that you dispatch work to a different thread when you're done with it in your pipeline.

Bootstrap

While it is possible to configure and register Channels with EventLoops directly, it is generally more useful to have a higher-level abstraction to handle this work.

For this reason, SwiftNIO ships a number of Bootstrap objects whose purpose is to streamline the creation of channels. Some Bootstrap objects also provide other functionality, such as support for Happy Eyeballs for making TCP connection attempts.

Currently SwiftNIO ships with three Bootstrap objects in the NIOPosix module: ServerBootstrap, for bootstrapping listening channels; ClientBootstrap, for bootstrapping client TCP channels; and DatagramBootstrap for bootstrapping UDP channels.

ByteBuffer

The majority of the work in a SwiftNIO application involves shuffling buffers of bytes around. At the very least, data is sent and received to and from the network in the form of buffers of bytes. For this reason it's very important to have a high-performance data structure that is optimized for the kind of work SwiftNIO applications perform.

For this reason, SwiftNIO provides ByteBuffer, a fast copy-on-write byte buffer that forms a key building block of most SwiftNIO applications. This type is provided by the NIOCore module.

ByteBuffer provides a number of useful features, and in addition provides a number of hooks to use it in an "unsafe" mode. This turns off bounds checking for improved performance, at the cost of potentially opening your application up to memory correctness problems.

In general, it is highly recommended that you use the ByteBuffer in its safe mode at all times.

For more details on the API of ByteBuffer, please see our API documentation, linked below.

Promises and Futures

One major difference between writing concurrent code and writing synchronous code is that not all actions will complete immediately. For example, when you write data on a channel, it is possible that the event loop will not be able to immediately flush that write out to the network. For this reason, SwiftNIO provides EventLoopPromise<T> and EventLoopFuture<T> to manage operations that complete asynchronously. These types are provided by the NIOCore module.

An EventLoopFuture<T> is essentially a container for the return value of a function that will be populated at some time in the future. Each EventLoopFuture<T> has a corresponding EventLoopPromise<T>, which is the object that the result will be put into. When the promise is succeeded, the future will be fulfilled.

If you had to poll the future to detect when it completed that would be quite inefficient, so EventLoopFuture<T> is designed to have managed callbacks. Essentially, you can hang callbacks off the future that will be executed when a result is available. The EventLoopFuture<T> will even carefully arrange the scheduling to ensure that these callbacks always execute on the event loop that initially created the promise, which helps ensure that you don't need too much synchronization around EventLoopFuture<T> callbacks.

Another important topic for consideration is the difference between how the promise passed to close works as opposed to closeFuture on a Channel. For example, the promise passed into close will succeed after the Channel is closed down but before the ChannelPipeline is completely cleared out. This will allow you to take action on the ChannelPipeline before it is completely cleared out, if needed. If it is desired to wait for the Channel to close down and the ChannelPipeline to be cleared out without any further action, then the better option would be to wait for the closeFuture to succeed.

There are several functions for applying callbacks to EventLoopFuture<T>, depending on how and when you want them to execute. Details of these functions is left to the API documentation.

Design Philosophy

SwiftNIO is designed to be a powerful tool for building networked applications and frameworks, but it is not intended to be the perfect solution for all levels of abstraction. SwiftNIO is tightly focused on providing the basic I/O primitives and protocol implementations at low levels of abstraction, leaving more expressive but slower abstractions to the wider community to build. The intention is that SwiftNIO will be a building block for server-side applications, not necessarily the framework those applications will use directly.

Applications that need extremely high performance from their networking stack may choose to use SwiftNIO directly in order to reduce the overhead of their abstractions. These applications should be able to maintain extremely high performance with relatively little maintenance cost. SwiftNIO also focuses on providing useful abstractions for this use-case, such that extremely high performance network servers can be built directly.

The core SwiftNIO repository will contain a few extremely important protocol implementations, such as HTTP, directly in tree. However, we believe that most protocol implementations should be decoupled from the release cycle of the underlying networking stack, as the release cadence is likely to be very different (either much faster or much slower). For this reason, we actively encourage the community to develop and maintain their protocol implementations out-of-tree. Indeed, some first-party SwiftNIO protocol implementations, including our TLS and HTTP/2 bindings, are developed out-of-tree!

Documentation

Example Usage

There are currently several example projects that demonstrate how to use SwiftNIO.

To build & run them, run following command, replace TARGET_NAME with the folder name under ./Sources

swift run TARGET_NAME

For example, to run NIOHTTP1Server, run following command:

swift run NIOHTTP1Server

Getting Started

SwiftNIO primarily uses SwiftPM as its build tool, so we recommend using that as well. If you want to depend on SwiftNIO in your own project, it's as simple as adding a dependencies clause to your Package.swift:

dependencies: [
    .package(url: "https://github.com/apple/swift-nio.git", from: "2.0.0")
]

and then adding the appropriate SwiftNIO module(s) to your target dependencies. The syntax for adding target dependencies differs slightly between Swift versions. For example, if you want to depend on the NIOCore, NIOPosix and NIOHTTP1 modules, specify the following dependencies:

Swift 5.4 and newer (swift-tools-version:5.4)

dependencies: [.product(name: "NIOCore", package: "swift-nio"),
               .product(name: "NIOPosix", package: "swift-nio"),
               .product(name: "NIOHTTP1", package: "swift-nio")]

Using Xcode Package support

If your project is set up as an Xcode project and you're using Xcode 11+, you can add SwiftNIO as a dependency to your Xcode project by clicking File -> Swift Packages -> Add Package Dependency. In the upcoming dialog, please enter https://github.com/apple/swift-nio.git and click Next twice. Finally, select the targets you are planning to use (for example NIOCore, NIOHTTP1, and NIOFoundationCompat) and click finish. Now will be able to import NIOCore (as well as all the other targets you have selected) in your project.

To work on SwiftNIO itself, or to investigate some of the demonstration applications, you can clone the repository directly and use SwiftPM to help build it. For example, you can run the following commands to compile and run the example echo server:

swift build
swift test
swift run NIOEchoServer

To verify that it is working, you can use another shell to attempt to connect to it:

echo "Hello SwiftNIO" | nc localhost 9999

If all goes well, you'll see the message echoed back to you.

To work on SwiftNIO in Xcode, you can just open the Package.swift file in Xcode and use Xcode's support for SwiftPM Packages.

An alternative: using docker-compose

Alternatively, you may want to develop or test with docker-compose.

First make sure you have Docker installed, next run the following commands:

  • docker-compose -f docker/docker-compose.yaml run test

    Will create a base image with Swift runtime and other build and test dependencies, compile SwiftNIO and run the unit and integration tests

  • docker-compose -f docker/docker-compose.yaml up echo

    Will create a base image, compile SwiftNIO, and run a sample NIOEchoServer on localhost:9999. Test it by echo Hello SwiftNIO | nc localhost 9999.

  • docker-compose -f docker/docker-compose.yaml up http

    Will create a base image, compile SwiftNIO, and run a sample NIOHTTP1Server on localhost:8888. Test it by curl http://localhost:8888

  • docker-compose -f docker/docker-compose.yaml -f docker/docker-compose.2204.57.yaml run test

    Will create a base image using Ubuntu 22.04 and Swift 5.7, compile SwiftNIO and run the unit and integration tests. Files exist for other ubuntu and swift versions in the docker directory.

Developing SwiftNIO

Note: This section is only relevant if you would like to develop SwiftNIO yourself. You can ignore the information here if you just want to use SwiftNIO as a SwiftPM package.

For the most part, SwiftNIO development is as straightforward as any other SwiftPM project. With that said, we do have a few processes that are worth understanding before you contribute. For details, please see CONTRIBUTING.md in this repository.

Prerequisites

SwiftNIO's main branch is the development branch for the next releases of SwiftNIO 2, it's Swift 5-only.

To be able to compile and run SwiftNIO and the integration tests, you need to have a few prerequisites installed on your system.

macOS

  • Xcode 11.4 or newer, Xcode 12 recommended.

Linux

  • Swift 5.7 or newer from swift.org/download. We always recommend to use the latest released version.
  • netcat (for integration tests only)
  • lsof (for integration tests only)
  • shasum (for integration tests only)

Ubuntu 18.04

# install swift tarball from https://swift.org/downloads
apt-get install -y git curl libatomic1 libxml2 netcat-openbsd lsof perl

Fedora 28+

dnf install swift-lang /usr/bin/nc /usr/bin/lsof /usr/bin/shasum

Benchmarks

Benchmarks for swift-nio are in a separate Swift Package in the Benchmarks subfolder of this repository. They use the package-benchmark plugin. Benchmarks depends on the jemalloc memory allocation library, which is used by package-benchmark to capture memory allocation statistics. An installation guide can be found in the Getting Started article of package-benchmark. Afterwards you can run the benchmarks from CLI by going to the Benchmarks subfolder (e.g. cd Benchmarks) and invoking:

swift package benchmark

For more information please refer to swift package benchmark --help or the documentation of package-benchmark.

swift-nio's People

Contributors

agnosticdev avatar basthomas avatar carolinacass avatar compnerd avatar davidde94 avatar ddunbar avatar dnadoba avatar fabianfett avatar finagolfin avatar franzbusch avatar glbrntt avatar gmilos avatar gwynne avatar hassila avatar helje5 avatar ianpartridge avatar ktoso avatar lukasa avatar maxdesiatov avatar mordil avatar normanmaurer avatar peteradams-a avatar rnro avatar ser-0xff avatar shekhar-rajak avatar simonjbeaumont avatar tanner0101 avatar tigerpixel avatar tomerd avatar weissi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

swift-nio's Issues

Docker fails: `docker-compose -f docker/docker-compose.yaml up test`

Expected behavior

docker-compose -f docker/docker-compose.yaml up test runs the test suite.

Actual behavior

Building the docker image fails:

Step 26/41 : ARG version=4.0.3
 ---> Using cache
 ---> 98a838f71a66
Step 27/41 : RUN mkdir $HOME/.swift
 ---> Using cache
 ---> c1011b5f61d1
Step 28/41 : RUN wget -q https://swift.org/builds/swift-${version}-release/ubuntu1404/swift-${version}-RELEASE/swift-${version}-RELEASE-ubuntu14.04.tar.gz -O $HOME/swift.tar.gz
 ---> Running in 3c69231a1f55
ERROR: Service 'swift-nio' failed to build: The command '/bin/sh -c wget -q https://swift.org/builds/swift-${version}-release/ubuntu1404/swift-${version}-RELEASE/swift-${version}-RELEASE-ubuntu14.04.tar.gz -O $HOME/swift.tar.gz' returned a non-zero code: 4

Steps to reproduce

Run docker-compose -f docker/docker-compose.yaml up test

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeaPro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64
Docker version 17.12.0-ce, build c97c6d6

P.S.: Can we just use the official image please?

channelRead invoked before channelActive

Expected behavior

The Channel callbacks lifecycle should start with channelActive and end with channelInactive.

Actual behavior

Sometimes under high concurrency the channelRead is invoked prior to channelActive.

Steps to reproduce

Run the attached code.

The code fails in one of three ways:

  1. It either gets stuck
  2. Or it fails with Illegal Instruction on a DispatchQueue (entering some q.sync)
  3. Or it fails with
Precondition failed: channelRead before channelActive!: file StuckTest/RemoteSystem.swift, line 224
Illegal instruction: 4

The first two failures might not be relevant, but it warrants to take a look. This particular issue is mainly about failure number 3, Precondition failed.

Minimal yet complete reproducer code

StuckTest.zip

SwiftNIO version/commit hash

681ddd8bab4fac997cbc70ca15a1e13c74da50b9 (nmaurer/order_fix)

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.1 (swiftlang-902.0.38 clang-902.0.30)
Target: x86_64-apple-darwin17.4.0
Darwin Levs-MacBook-Pro.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

consider exposing the current `EventLoop`

In certain cases nicer APIs can be created if there's a public way to get the current EventLoop from within a SwiftNIO app. We should consider doing that. The most straight-forward implementation is obviously to just store in in a ThreadSpecificVariable.

Type-safe ChannelPipeline

It would be great to have a type-safe ChannelPipeline. The whole ChannelPipeline can't be all type-safe as we want to support arbitrary modifications of it at runtime (for example for HTTP/1 to HTTP/2 upgrades).
However it would be great to be able to have at least a partially type-safe pipeline which could be fully type-safe for programs who don't need the dynamism.

think about if we want easier, more high-level byte writing APIs

suggested by @ddunbar

SwiftNIO might be appealing to developers of network servers that need good but maybe not the best network performance. In these cases, the allocate ByteBuffer, accumulate, ctx.write, ..., ctx.flush might be a bit awkward. A simpler API might also be a good starting point. For ChannelHandlers with OutboundOut = ByteBuffer, @ddunbar proposed the following API:

ctx.writeWithStream { stream in
    stream <<< “this” <<< Format.asJSON(…) <<< “bla bla bla”
}

which would basically do:

{
    var buf = ctx.channel.allocate(capacity: 256 /* random guess */)
    buf.write(staticString: "this")
    buf.write(bytes: Format.asJSON(...)) // Format.asJSON is actually more sophisticated than that. It returns a "formatter", which is a struct which knows how to render the output as JSON directly onto the stream. This avoids creating unnecessary temporary allocations of the JSON data.
    buf.write(staticString: "bla bla bla")
    return ctx.write(self.wrapOutboundOut(.byteBuffer(buf)))
}()

and I guess we could also offer a ctx.writeAndFlushWithStream API. Or maybe ctx.writeWithStream(flush: Bool = false, (Stream) -> EventLoopPromise<()>) -> EventLoopPromise<()> 🤔.

The custom <<< operator would be defined for everything we accept: ByteBuffer, (Contiguous)Collection of UInt8, String, StaticString, the various integers, ...

We should definitely evaluate having such a more high-level API.

This should probably live in some yet to be created NIOExtras module. And we need to think about how to make it perform well and in an unsurprising way.

Channel writability thresholds work weirdly.

The current mode for using channel writability thresholds looks like this:

try firstChannel.setOption(option: ChannelOptions.writeBufferWaterMark, value: 0..<1024)

Under the covers this uses .upperBound and .lowerBound to manage the writability. Specifically, the channel is marked "not writable" when the buffered byte count is > .upperBound, and marked writable again when it drops < .lowerBound.

I found this behaviour to be moderately surprising for two reasons. Firstly, there's this:

  1> (0..<1024).upperBound
$R0: Int = 1024
  2> (0...1024).upperBound
$R1: Int = 1024

This means that for either a closed or semi-open range Swift reports the same upper bound. Frankly this behaviour is so surprising I kinda want to call it a bug in Swift, but as a user I'd expect that if I typed 0..<1024 that we'd mark the channel not writable when we write the 1024'th byte, not the 1025th.

The bigger problem is that in both of these cases, once the channel is marked not writable it will never be marked writable again. This is definitely bad, and if we think that behaviour makes sense for this kind of range then we should forbid ranges that are zero-bounded at the bottom. However, I'd argue that we should do a <= check instead of a < check for the lower bound.

Alternatively, we may just want to chuck this slightly confusing interface altogether and instead split the option into two: a lower threshold and an upper one. We can then define the semantic for each threshold directly, rather than have users see a range and expect they know what it does.

Regardless, at the barest minimum we should either do some sanity checking of the range the user passes us or change the type definition to Range<UInt> because right now we also allow this:

try firstChannel.setOption(option: ChannelOptions.writeBufferWaterMark, value: -100..<(-50))

And that's obviously totally 🍌.

reduce indirection for channel addresses

currently we have

private let localAddressCached: AtomicBox<Box<SocketAddress?>> = AtomicBox(value: Box(nil))

which is quite bad as it's a box in a box. Given that SocketAddress already has its private box, we might want to make that private box internal and use it here :).

Btw, AtomicBox also has one level of indirection more than it should have.

Missing remoteAddress in ChatServer (Crash)

Expected behavior

When the ChatClient sends a message, other clients see

(##remoteAddress##) - ##message##

Actual behavior

The ChatServer force-unwraps the remoteAddress field, encountering nil, and crashes.

Steps to reproduce

  1. Run ChatServer
  2. Run ChatClient (once is all that's needed to demonstrate crash, twice for full set up)
  3. Send message by typing in ChatClient and pressing enter
  4. ChatServer unexpectedly encounters nil here:
    buffer.write(string: "(\(ctx.channel.remoteAddress!)) - ")

Workaround

Modifying the above line referenced in part 4 to the following produces what appears to be expected behavior. However, I am unclear as to the difference between ctx.channel.remoteAddress and ctx.remoteAddress.

if let remoteAddress = ctx.remoteAddress {
    buffer.write(string: "(\(remoteAddress)) - ")
} else {
    buffer.write(string: "(no address) - ")
}

If possible, minimal yet complete reproducer code (or URL to code)

Provided ChatServer and ChatClient exhibit this behavior

SwiftNIO version/commit hash

2c1d993 (latest as this issue)

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin Ezekiels-MacBook-Pro-133.local 17.5.0 Darwin Kernel Version 17.5.0: Sun Feb 11 23:54:24 PST 2018; root:xnu-4570.50.279~25/RELEASE_X86_64 x86_64

Yep

Expected behavior

[what you expected to happen]

Actual behavior

[what actually happened]

Steps to reproduce

  1. ...
  2. ...

If possible, minimal yet complete reproducer code (or URL to code)

[anything to help us reproducing the issue]

SwiftNIO version/commit hash

[the SwiftNIO tag/commit hash]

Swift & OS version (output of swift --version && uname -a)

Spanish translation

I'm interested in beginning translating the project into Spanish and I was wondering if I could start pull requests of the files. Many thanks in advance!

Compiling NIO in Release Mode throws warnings

Expected behavior

No warnings, clean compile.

Actual behavior

A set of warnings, no clean compile.

Compile Swift Module 'NIO' (47 sources)
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:316:25: warning: will never be executed
            if let me = eventLoop as? SelectableEventLoop {
                        ^
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:315:12: note: condition always evaluates to false
        if _isDebugAssertConfiguration() {
           ^
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:338:64: warning: will never be executed
        if _isDebugAssertConfiguration(), let eventLoop = self.eventLoop as? SelectableEventLoop {
                                                               ^
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/swift-nio.git-3108475404973543938/Sources/NIO/EventLoopFuture.swift:338:12: note: condition always evaluates to false
        if _isDebugAssertConfiguration(), let eventLoop = self.eventLoop as? SelectableEventLoop {
           ^
Compile Swift Module 'NIOTLS' (3 sources)
Compile Swift Module 'NIOHTTP1' (7 sources)
Compile Swift Module 'NIOOpenSSL' (11 sources)
Compile Swift Module 'MicroExpress' (7 sources)
/Users/helge/dev/Swift/SwiftXcode/Images/SwiftNIO_XcodeImage/.build/checkouts/MicroExpress.git-6340581023861930988/Sources/MicroExpress/Express.swift:23:26: warning: 'addHTTPServerHandlers(first:)' is deprecated: Please use configureHTTPServerPipeline
        channel.pipeline.addHTTPServerHandlers().then {
                         ^

Steps to reproduce

swift build -c release

SwiftNIO version/commit hash

1.2.1

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeaPro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64

ChannelInvoker.newPromise should pass file and line to eventLoop.newPromise

We normally want debugging information to keep track of where our promises are allocated in debug modes. However, if someone does Channel.newPromise instead of eventLoop.newPromise, that information is lost. The core issue is here:

private func newPromise() -> EventLoopPromise<Void> {
return eventLoop.newPromise()
}

This indirection should preserve the #file and #line specific debugging logic we use elsewhere.

Improve EventLoopFuture code

A lot of the EventLoopFuture code is quite prone to subtle correctness bugs. This happens for a bunch of reasons, most of them relating to the way the code has grown over time. While we've patched over them a bit, we should really take a step to clean this code up more profoundly, ideally one method at a time.

Here are some things we should aim to do:

  • Make map faster.

    Right now map performs worse than many users would expect. This is because map is basically then with a call to eventLoop.newSucceededFuture. This means that a call to map forces allocation of a new EventLoopPromise and a new EventLoopFuture. While this is appealing from the perspective of correctness ("well, map is really just the combination of monadic uplift and bind, blah blah blah haskell") it discourages our internal code from using it for performance reasons.

    This allocation is fundamentally unnecessary, and a better internal API could draw a distinction between these two behaviours to improve the performance of map.

  • Define a better internal API.

    Quite a few methods try to be fast by skipping the external API, not least because map is slow. This is problematic, because the internal API exposes all the warty insides of these types ("what's a CallbackList? Why do I need to return one? How is then implemented, and how do I approximate it?"). We should clean these interfaces up to make it easier to be confident that the code that uses them is correct. In particular, the way they interact with the thread-hopping behaviour of EventLoopFuture needs to be made extremely clear, so that we can validate that code that uses them is actually safe.

  • More utility functions.

    While we have and<T, U> -> EventLoopFuture<(T, U)> and andAll<Void> -> EventLoopFuture<Void>, there are a few other primitives we should probably provide. In particular, it's easy enough to extend andAll<Void> to andAll<T> -> EventLoopFuture<[T]>, as well as provide a variant of andAll that fails slow instead of failing fast (e.g. collate<T> -> EventLoopFuture<[Result<T>]>).

Any other suggestions?

ChannelNotificationTest.testActiveBeforeChannelRead is flaky.

Blocking the merge of #191:

15:41:06 Test Case 'ChannelNotificationTest.testActiveBeforeChannelRead' started at 2018-03-20 14:41:06.816
15:41:06 Fatal error: leaking promise created at (file: "/code/Tests/NIOTests/ChannelNotificationTest.swift", line: 377): file /code/Tests/NIOTests/ChannelNotificationTest.swift, line 377
15:41:06 Current stack trace:
15:41:06 0    libswiftCore.so                    0x00007f5a5c3d4f90 _swift_stdlib_reportFatalErrorInFile + 221
15:41:06 1    libswiftCore.so                    0x00007f5a5c0de330 closure #1 in closure #1 in closure #1 in _assertionFailure(_:_:file:line:flags:) + 284
15:41:06 2    libswiftCore.so                    0x00007f5a5c37e9a1 <unavailable> + 4155809
15:41:06 3    libswiftCore.so                    0x00007f5a5c395059 <unavailable> + 4247641
15:41:06 4    libswiftCore.so                    0x00007f5a5c0ddc0b <unavailable> + 1399819
15:41:06 5    libswiftCore.so                    0x00007f5a5c2f76b9 <unavailable> + 3602105
15:41:06 6    libswiftCore.so                    0x00007f5a5c33579f <unavailable> + 3856287
15:41:06 7    libswiftCore.so                    0x00007f5a5c0ddc0b <unavailable> + 1399819
15:41:06 8    libswiftCore.so                    0x00007f5a5c2777a0 specialized _assertionFailure(_:_:file:line:flags:) + 144
15:41:06 9    swift-nioPackageTests.xctest       0x00000000004b5dac <unavailable> + 744876
15:41:06 10   swift-nioPackageTests.xctest       0x00000000004b5f0c <unavailable> + 745228
15:41:06 11   swift-nioPackageTests.xctest       0x0000000000557ebc <unavailable> + 1408700
15:41:06 12   swift-nioPackageTests.xctest       0x0000000000557dbd <unavailable> + 1408445
15:41:06 13   swift-nioPackageTests.xctest       0x00000000004b5951 <unavailable> + 743761
15:41:06 14   swift-nioPackageTests.xctest       0x00000000004b5f3f <unavailable> + 745279
15:41:06 15   libswiftCore.so                    0x00007f5a5c39c1cb <unavailable> + 4276683
15:41:06 16   swift-nioPackageTests.xctest       0x000000000075a841 <unavailable> + 3516481
15:41:06 17   swift-nioPackageTests.xctest       0x000000000075e0a2 <unavailable> + 3530914
15:41:06 18   swift-nioPackageTests.xctest       0x00000000004414c6 <unavailable> + 267462
15:41:06 19   swift-nioPackageTests.xctest       0x000000000075e15b <unavailable> + 3531099
15:41:06 20   libXCTest.so                       0x00007f5a5c859c7f <unavailable> + 203903
15:41:06 21   libXCTest.so                       0x00007f5a5c857721 <unavailable> + 194337
15:41:06 22   libXCTest.so                       0x00007f5a5c85996f <unavailable> + 203119
15:41:06 23   libXCTest.so                       0x00007f5a5c8598ab <unavailable> + 202923
15:41:06 24   libXCTest.so                       0x00007f5a5c85b119 <unavailable> + 209177
15:41:06 25   libXCTest.so                       0x00007f5a5c85a5c0 <unavailable> + 206272
15:41:06 26   libXCTest.so                       0x00007f5a5c83d800 XCTestCase.invokeTest() + 102
15:41:06 27   libXCTest.so                       0x00007f5a5c84dea6 <unavailable> + 155302
15:41:06 28   libXCTest.so                       0x00007f5a5c83d740 XCTestCase.perform(_:) + 14
15:41:06 29   libXCTest.so                       0x00007f5a5c837f20 XCTest.run() + 662
15:41:06 30   libXCTest.so                       0x00007f5a5c84e77b <unavailable> + 157563
15:41:06 31   libXCTest.so                       0x00007f5a5c84e6b0 <unavailable> + 157360
15:41:06 32   libXCTest.so                       0x00007f5a5c84ea84 <unavailable> + 158340
15:41:06 33   libXCTest.so                       0x00007f5a5c83a3b0 XCTMain(_:) + 3770
15:41:06 34   swift-nioPackageTests.xctest       0x00000000009c61ed <unavailable> + 6054381
15:41:06 35   libc.so.6                          0x00007f5a5a269e50 __libc_start_main + 245
15:41:06 36   swift-nioPackageTests.xctest       0x000000000041b089 <unavailable> + 110729

IPv6 Address description contains 0-bytes

Expected behavior

No 0 bytes in strings

Actual behavior

0 bytes in strings

Steps to reproduce

In Xcode:

  1. after a server bootstrap, doprint("Server running on:", serverChannel.localAddress!)
  2. copy output to e.g. TextMate

It says:
Server running on: [IPv6]::1<NUL><NUL><NUL>....lots...

Some editors seem to eat the \0's.

Cause

I think it is

private func descriptionForAddress(family: CInt, bytes: UnsafeRawPointer, length byteCount: Int) -> String {
...
            return String(decoding: UnsafeBufferPointer<UInt8>(start: addressBytesPtr, count: byteCount), as: Unicode.ASCII.self)

That leaves the 0's from the address buffer in the String. I think you probably want some

return String(cString: addressBytesPtr)

variant here, so that the string terminates at the \0.

Other potential causes: incorrect indentation, 80-column source line length overflows all over.

SwiftNIO version/commit hash

1.1

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeaPro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64 x86_64

fatalError() during high concurrency test

Expected behavior

Users generally expect Swift NIO not to fail with internal assertions and fatalErrors they couldn't catch.

Actual behavior

Fatal error: Invalid FSM transition attempt: state complete, input connectTimeoutElapsed: file swift-nio/Sources/NIO/HappyEyeballs.swift, line 417
Illegal instruction: 4

Steps to reproduce

Compile and run the attached code against the master.

If possible, minimal yet complete reproducer code (or URL to code)

reproduce.swift.

SwiftNIO version/commit hash

b8d883da2c2090d02ec4890b5ab5c7c9bfa30935

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.1 (swiftlang-902.0.38 clang-902.0.30)
Target: x86_64-apple-darwin17.4.0
Darwin Levs-MacBook-Pro.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

Write narrative documentation to explain what Inbound/Outbound mean

Given that channelRead is the most prominent ChannelInboundHandler method, and that write is the most prominent ChannelOutboundHandler method, it's easy for users to assume that inbound means reads, and outbound means writes. This will cause user confusion when they discover that channelWritabilityChanged is an Inbound method, and that read is an Outbound one!

We should write some narrative docs for these two protocols to make it clear that this is not actually how they work: instead, that inbound/outbound is about causality. Specifically, Inbound events are things the socket tells the pipeline, while Outbound events are things the pipeline tells the socket. From that construction it becomes clear why read is Outbound (the pipeline tells the socket to do a read), and why channelWritabilityChanged is Inbound (the socket tells the pipeline whether it's writable or not).

Document differences to Netty

While SwiftNIO is more or fairly close to Netty 4.1 there are a few things which differ (in many cases things we want to fix in Netty 5.x as well). We should document these differences.

What are the plans to support macOS 10.11 and iOS?

I am interested in using SwiftNIO's HTTP2 implementation (once available) to add an NIO-based server implementation to https://github.com/grpc/grpc-swift. Given that that project currently depends on several other fairly large libraries, it would be great to also move the client code to SwiftNIO to eliminate those dependencies.

For that to happen, though, SwiftNIO would need to support those client platforms — namely iOS (preferably 10+) and macOS 10.11. Neither currently shows up in the "Supported Platforms" list — does that mean it's technically impossible to support those platforms (e.g. because some low-level networking APIs are missing), or has the library simply not been tested on those platforms, but might already work as-is? Other than that, what level of effort would be required to add support for these two platforms, or is it already planned, and with what timeline?

we should query getaddrinfo more precisely

Expected behavior

When asking getaddrinfo about an IP address, we should not get two results that contain the same IP address.

Actual behavior

we do get two, example here:

(lldb) p info.pointee
(addrinfo) $R13 = {
  ai_flags = 0
  ai_family = 2
  ai_socktype = 2
  ai_protocol = 17
  ai_addrlen = 16
  ai_canonname = nil
  ai_addr = 0x0000000104904ab0
  ai_next = 0x0000000104904f60
}
(lldb) p info.pointee.ai_next.pointee
(addrinfo) $R14 = {
  ai_flags = 0
  ai_family = 2
  ai_socktype = 1
  ai_protocol = 6
  ai_addrlen = 16
  ai_canonname = nil
  ai_addr = 0x0000000104904f90
  ai_next = nil
}

whereby ai_protocol = 17 is IPPROTO_UDP and ai_protocol = 6 is IPPROTO_TCP. But since we know which one we want we should query correctly.

Steps to reproduce

  • connect to say 127.0.0.1
  • set a break point at the last line of private func parseResults(_ info: UnsafeMutablePointer<addrinfo>, host: String)

If possible, minimal yet complete reproducer code (or URL to code)

bootstrap.connect(host: "127.0.0.1", port: 1234)

SwiftNIO version/commit hash

1.2.0

Swift & OS version (output of swift --version && uname -a)

$ uname -a 
Darwin jwmbp.local 17.5.0 Darwin Kernel Version 17.5.0: Mon Jan 22 18:03:58 PST 2018; root:xnu-4570.50.252~1/DEVELOPMENT_X86_64 x86_64
$ swift --version
Apple Swift version 4.0.3 (swiftlang-900.0.71 clang-900.0.38)
Target: x86_64-apple-macosx10.9

One Failing Integration Test

Expected behavior

All tests in the integration Test Suite to pass.

Actual behavior

1 Failing integration test:

test_16_tcp_client_ip.sh

Logs:

Running test 'test_16_tcp_client_ip.sh'... FAILURE (0)
--- OUTPUT BEGIN ---
+ set -o pipefail
+ test=/Users/martinho/Development/swift-nio/IntegrationTests/tests_01_http/test_16_tcp_client_ip.sh
+ tmp=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t
+ root=/Users/martinho/Development/swift-nio/IntegrationTests/..
+++ dirname /Users/martinho/Development/swift-nio/IntegrationTests/run-single-test.sh
++ cd /Users/martinho/Development/swift-nio/IntegrationTests
++ pwd
+ here=/Users/martinho/Development/swift-nio/IntegrationTests
+ source /Users/martinho/Development/swift-nio/IntegrationTests/test_functions.sh
+ source /Users/martinho/Development/swift-nio/IntegrationTests/tests_01_http/test_16_tcp_client_ip.sh
++ source defines.sh
+++ create_token
+++ mktemp /tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_XXXXXX
++ token=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi
++ start_server /tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi tcp
++ local token=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/server_token_hmM3vi
++ local type=--uds
++ local port=/tmp/.swift-nio-http1-server-sh-tests_i4bwvi/test.tmp_oqB43t/port.sock
++ local tmp_server_pid
++ local tok_type=--unix-socket
++ local curl_port=80
++ maybe_host=
++ maybe_nio_host=
++ [[ tcp == \t\c\p ]]
++ type=
++ port=0
++ tok_type=
++ maybe_host=localhost
+++ host -4 -t a localhost
+++ tr ' ' '\n'
+++ tail -1
++ maybe_nio_host='3(NXDOMAIN)'
--- OUTPUT  END  ---

Steps to reproduce

  1. Run: swift build
  2. Run: swift test
  3. Run: scripts/integration_tests.sh

SwiftNIO version/commit hash

1.0.0 / 5ff6d9b

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin Tiagos-MacBook-Pro-2.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

Performance improvements for HTTP

Currently the HTTP performance is not as good as we have hoped for.... This may be caused by multiple things:

  • cross-module calls (no optimisations for this in swift yet :( )
  • HttpHeaders create Strings in a non-lazy fashion
  • others....

We should look into what we can do to make it faster.

HTTPMethod.hasRequestBody `.unlikely`

The current implementation of hasRequestBody uses unlikely for a lot of cases where a body is actually very likely or a plain .yes (not sure what the different is, a PUT can also have an empty body).

Examples MKCOL, PROPFIND, REPORT, MKCALENDAR, etc.

One Failing Test Case on Ubuntu

Insight

I saw that in the Docker configuration multiple versions of Swift are defined (4.0.2 & 4.0). There is a particular reason for it @tomerd?
Upgrading it to the latest version (4.0.3) makes all the Test Suite pass on Ubuntu.
Should I do directly a PR with these changes?

Expected behavior

All tests in the Test Suite to pass.

Actual behavior

1 Failing Test Case:

HTTPUpgradeTestCase.testDelayedUpgradeBehaviour

Logs:

test_1       | Test Case 'HTTPUpgradeTestCase.testDelayedUpgradeBehaviour' started at 2018-03-03 03:28:58.078
test_1       | /code/Tests/NIOHTTP1Tests/HTTPUpgradeTests.swift:694: error: HTTPUpgradeTestCase.testDelayedUpgradeBehaviour : XCTAssertFalse failed -
test_1       | Test Case 'HTTPUpgradeTestCase.testDelayedUpgradeBehaviour' failed (0.003 seconds)

Steps to reproduce

  1. Install Docker for Mac
  2. Run: cd docker
  3. Run: docker-compose up test

If possible, minimal yet complete reproducer code (or URL to code)

Just follow the README instructions on how to run the test suite for Ubuntu using Docker.

SwiftNIO version/commit hash

1.0.0 / 5ff6d9b

Swift & OS version (output of swift --version && uname -a)

Specified in the Docker configuration.

Too many calls to futex

Expected behavior

I expect there not to be so many futex calls.

Actual behavior

916   20:12:30 epoll_create(128)        = 3 <0.000055>                                                                                                                         
916   20:12:30 futex(0x667b10, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000023>
916   20:12:30 futex(0x667b20, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000033>
916   20:12:30 eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = 4 <0.000064>
916   20:12:30 futex(0x667af0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000032>
916   20:12:30 futex(0x667b00, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000022>
916   20:12:30 timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC|TFD_NONBLOCK) = 5 <0.000058>
916   20:12:30 futex(0x667b60, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000022>
916   20:12:30 futex(0x667b80, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000033>
916   20:12:30 futex(0x667b90, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000032>
916   20:12:30 futex(0x667b30, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000023>
916   20:12:30 epoll_ctl(3, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLERR|EPOLLRDHUP, {u32=4, u64=4}}) = 0 <0.000068>
916   20:12:30 futex(0x667ba0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000020>
916   20:12:30 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLERR|EPOLLRDHUP|EPOLLET, {u32=5, u64=5}}) = 0 <0.000053>
916   20:12:30 futex(0x6686c8, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000068>
916   20:12:30 futex(0x6680b0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000056>
916   20:12:30 futex(0x7f81323e9530, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000064>
916   20:12:30 futex(0x7f81323e9928, FUTEX_WAKE_PRIVATE, 2147483647) = 0 <0.000068>

Steps to reproduce

  1. docker run --security-opt seccomp:unconfined -v$CWD:/code -it --rm swiftnio
  2. cd /code && swift build
  3. strace -s 1024 -fF -tT -o trace.out ./.build/x86_64-unknown-linux/debug/NIOHTTP1Server 0.0.0.0 8888 .

If possible, minimal yet complete reproducer code (or URL to code)

There are no futex calls here:

root@1ba3a0b6d2c0:/code# cat epoll.c 
#include <sys/epoll.h>

int main() {
    int fd = epoll_create(1);
    epoll_wait(fd, 0, 0, 0);
    return 0;
}
root@1ba3a0b6d2c0:/code# clang epoll.c 
root@1ba3a0b6d2c0:/code# strace -s 1024 -f -tT ./a.out 
21:46:15 execve("./a.out", ["./a.out"], [/* 11 vars */]) = 0 <0.001591>
21:46:15 brk(NULL)                      = 0x1c96000 <0.000028>
21:46:15 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) <0.000200>
21:46:15 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000012>
21:46:15 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 <0.000253>
21:46:15 fstat(3, {st_mode=S_IFREG|0644, st_size=18870, ...}) = 0 <0.000017>
21:46:15 mmap(NULL, 18870, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f74ce3e6000 <0.000023>
21:46:15 close(3)                       = 0 <0.000015>
21:46:15 access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory) <0.000036>
21:46:15 open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 <0.000022>
21:46:15 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0@\0\0\0\0\0\0\0\270r\34\0\0\0\0\0\0\0\0\0@\0008\0\n\0@\0H\0G\0\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0`f\31\0\0\0\0\0`f\31\0\0\0\0\0`f\31\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\373\33\0\0\0\0\0\20\373\33\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\300\7\34\0\0\0\0\0\300\7<\0\0\0\0\0\300\7<\0\0\0\0\0`O\0\0\0\0\0\0\340\221\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\240;\34\0\0\0\0\0\240;<\0\0\0\0\0\240;<\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0\300\7\34\0\0\0\0\0\300\7<\0\0\0\0\0\300\7<\0\0\0\0\0\20\0\0\0\0\0\0\0x\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0P\345td\4\0\0\0|f\31\0\0\0\0\0|f\31\0\0\0\0\0|f\31\0\0\0\0\0\274T\0\0\0\0\0\0\274T\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0\300\7\34\0\0\0\0\0\300\7<\0\0\0\0\0\300\7<\0\0\0\0\0@8\0\0\0\0\0\0@8\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\2658\32Ey\6\322y\0078\"\245\316\262LK\376\371M\333\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\363\3\0\0\n\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1\210\3\346\220\305E\214\0\304\0\10\0\5\204\0`\300\200\0\r\212\f\0\4\20\0\210@2\10*@\210T<, \0162H&\204\300\214\4\10\0\2\2\16\241\254\32\4f\300\0\3002\0\300\0P\1 \201\10\204\v  ($\0\4 Z\0\20X\200\312DB(\0\6\200\20\30B\0 @\200\0IP\0Q\212@\22\0\0\0\0\10\0\0\21\20", 832) = 832 <0.000024>
21:46:15 fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0 <0.000013>
21:46:15 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f74ce3e5000 <0.000111>
21:46:15 mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f74cddfc000 <0.000057>
21:46:15 mprotect(0x7f74cdfbc000, 2097152, PROT_NONE) = 0 <0.000041>
21:46:15 mmap(0x7f74ce1bc000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7f74ce1bc000 <0.000044>
21:46:15 mmap(0x7f74ce1c2000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f74ce1c2000 <0.000019>
21:46:15 close(3)                       = 0 <0.000013>
21:46:15 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f74ce3e4000 <0.000020>
21:46:15 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f74ce3e3000 <0.000011>
21:46:15 arch_prctl(ARCH_SET_FS, 0x7f74ce3e4700) = 0 <0.000159>
21:46:15 mprotect(0x7f74ce1bc000, 16384, PROT_READ) = 0 <0.000030>
21:46:15 mprotect(0x600000, 4096, PROT_READ) = 0 <0.000068>
21:46:15 mprotect(0x7f74ce3eb000, 4096, PROT_READ) = 0 <0.000024>
21:46:15 munmap(0x7f74ce3e6000, 18870)  = 0 <0.000019>
21:46:15 epoll_create(1)                = 3 <0.000044>
21:46:15 epoll_wait(3, NULL, 0, 0)      = -1 EINVAL (Invalid argument) <0.000015>
21:46:15 exit_group(0)                  = ?
21:46:15 +++ exited with 0 +++

SwiftNIO version/commit hash

c8d1980

Swift & OS version (output of swift --version && uname -a)

Swift version 4.0 (swift-4.0-RELEASE)
Target: x86_64-unknown-linux-gnu
Linux 1ba3a0b6d2c0 4.9.60-linuxkit-aufs #1 SMP Mon Nov 6 16:00:12 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

`channelRead` is triggered w/ autoRead = false

Expected behavior

No channelRead is triggered on the inbound handler if autoRead is off, and the outbound handler does no read() request.

Actual behavior

After connecting to a server socket, one read seems to happen regardless of the autoRead option:

  .childChannelOption(ChannelOptions.autoRead, value: false)

Subsequent input is not reported (just the first stuff entered in nc).

Steps to reproduce

I have no minimal reproduce setup yet. Lazy.

SwiftNIO version/commit hash

1.1

Swift & OS version (output of swift --version && uname -a)

4.0.3 , macOS

Use official Swift Docker image

Expected behavior

Swift NIO reuses the official Swift Docker image instead of brewing its own soup.

Actual behavior

The Swift NIO Docker builds its own environment, with all the dependencies etc.

Steps to reproduce

docker build

First of all, people w/ Docker on the Mac, quite likely already have the image downloaded and can get started quicker.
Second, you use a consistent environment for Setty which other people already use. You'll catch issues you may not be seeing.

I don't know if you need anything extra to what the official image provides. If not, it would also result in a reproducible build/test environment. (If yes, you may want to push an own repo based on swift:4.0.3).

You can find it over here, it even says "OFFICIAL REPOSITORY" 😁: https://hub.docker.com/_/swift/

Thoughts on gRPC

Thank you for this library, this is some great work.
I was wondering if there is any plan to work on a gRPC library in Swift, built on top of swift-nio. I'd be more than interested to contribute and wanted to check if gRPC was part of some roadmap, directly (or undirectly) related to swift-nio and this project.
Thank you.

ByteToMessageDecoder behaves gracelessly when channelInactive is triggered while decode() is on the stack.

Expected behavior

When a ByteToMessageDecoder calls ctx.close() during its processing, and that close completes synchronously, receiving channelInactive will not cause ByteToMessageDecoder to re-parse previously parsed data.

Actual behavior

When this happens, ByteToMessageDecoder.channelInactive calls decodeLast with a fresh copy self.cumulationBuffer. By default, decodeLast is the same as decode, so this essentially calls decode in almost all cases. This means that the call stack ends up looking like this:

  1. decode()
  2. ctx.close()
    ...
  3. decodeLast()
  4. decode()

Worse, this second call to decode does it with an out of date copy of the cumulationBuffer, which it necessarily must do because we always operate on a copy of the cumulationBuffer and save it off after our calls to decode.

If possible, minimal yet complete reproducer code (or URL to code)

final class BufferLengthReporter: ByteToMessageDecoder {
    typealias InboundIn = ByteBuffer
    typealias InboundOut = Int

    var cumulationBuffer: ByteBuffer?

    func decode(ctx: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
        ctx.fireChannelRead(self.wrapInboundOut(buffer.readableBytes))
        buffer.moveReaderIndex(forwardBy: 4)
        ctx.close(promise: nil)
        return .needMoreData
    }

    func decodeLast(ctx: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
        ctx.fireChannelRead(self.wrapInboundOut(buffer.readableBytes))
        return .needMoreData
    }
}

class Test: XCTestCase {
    func testDontRepeatedlyParse() throws {
        let channel = EmbeddedChannel()
        defer {
            _ = try? channel.finish()
        }

        XCTAssertNoThrow(try channel.pipeline.add(handler: BufferLengthReporter()).wait())

        var buffer = channel.allocator.buffer(capacity: 5)
        buffer.write(staticString: "hello")

        XCTAssertTrue(try channel.writeInbound(buffer))
        XCTAssertFalse(channel.isActive)

        XCTAssertEqual(5, channel.readInbound())
        XCTAssertEqual(1, channel.readInbound())
        XCTAssertEqual(nil as Int?, channel.readInbound())
    }
}

The second XCTAssertEqual in the final block fails, reporting 5 instead of 1.

SwiftNIO version/commit hash

1.1.0

Possible Fixes

1. Remove decodeLast.

decodeLast is a strange function. It exists for protocols for whom EOF is a meaningful signal, but those protocols are not that common. More importantly, in the case of those protocols they could simply implement channelInactive themselves to handle the case. Right now, for the default protocol case, decodeLast is simply a waste of CPU: the protocol couldn't decode the buffer before, no reason to assume it can now.

The advantage of removing decodeLast is that it becomes clear that this is a thing users need to handle themselves. The disadvantage is that it doesn't solve this problem for handlers that treat EOF as meaningful: they still need a way to work out whether they're synchronously receiving EOF from the remote peer.

2. Change the default behaviour of decodeLast.

A smaller-scale version of (1), this would just change decodeLast to be a no-op. This avoids the current problem, but still provides the hook. The downside of this is, again, we end up letting users fall back into this trap.

3. Manage state better in ByteToMessageDecoder.

This is probably my preferred option, but it makes B2MD substantially more complex. Ultimately the core issue is that B2MD will eagerly dispatch channelInactive, which means that it's partway through parsing when that message is processed. That's just a bad idea. So we could add state to B2MD to record that it has seen channelInactive, but to not dispatch it while we're inside decode. Instead, we'd let decode return and then spot the change, and transition to calling decodeLast.

This is likely to behave better in the long-term.

4. Prevent channelInactive firing synchronously.

This is not a great fix, but most users do not realise that the moment they call out to third-party code they are at risk of being re-entrantly called by all kinds of events. We could, in principle, prevent close re-entrantly firing channelInactive, but frankly I don't think that's a very good idea. I just wanted to note it here.

All-in-all I think I prefer (3), but I'd like to hear what others think.

Add reference implementation for higher numbers of concurrent requests to HTTP1Server

It is likely I'm missing something, but here we go:

Expected behavior

I should be able to get more than 6 concurrent requests from NIOHTTP1Server

Actual behavior

I'm tracking concurrent requests like so:
https://gist.github.com/GeorgeLyon/df6f34c7260997f5c373ac9a1e81c9a7

If I spam the server with requests to /write-delay, my max stays at 6. I would like that to be much higher.

It seems to me that NIO is allowing 1 active channel per core only. Do I need to do something special to "suspend" the Channel for the duration of the long running task so as to let the next request through? I didn't see anything in ChannelOptions, ServerBootstrap or ChannelHandlerContext to adjust this.

SwiftNIO version/commit hash

1fdee50

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.1 (swiftlang-902.0.43 clang-902.0.37.1)
Target: x86_64-apple-darwin17.4.0
Darwin Georges-MacBook-Pro-2.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

`HTTPMethod` should not be an enum

Expected behavior

No API breakage if an HTTP method is added to the enum.

Actual behavior

API breakage if an HTTP method is added to the enum.

SwiftNIO version/commit hash

1.1

As long as Swift doesn't provide an open enum construct (which I think was in discussion, don't know the state of it), it is not wise to use a (closed) enum for constants which are unstable (in an API which is supposed to be stable). In this case, HTTP methods are added once in a while by new RFCs (I myself added MKCALENDAR to http_parser).

This was also discussed a lot in the HTTP WG effort, where we finally ended up with the common iOS solution: https://github.com/swift-server/http/blob/develop/Sources/HTTP/HTTPMethod.swift#L10 (I'm not sure this is the best solution, probably not, but it at least provides better API stability).

In short: Adding a new case to HTTPMethod currently requires a major SemVer bump. That is non-ideal as the addition of a HTTP methods is otherwise not a big event / breaking change ;-)

it's too hard to make HTTP/1.1 pipelining work correctly, we should offer a solution

At the moment, it's too hard to make HTTP/1.1 pipelining work correctly. SwiftNIO just forwards you the HTTP request parts as they come in and the user needs to make sure everything works correctly with pipelining which is hard (even our example NIOHTTP1Server doesn't do that (see #9)).

We should offer a HTTP1PipeliningHandler which forwards the n-th request only if the (n-1)th request has been completed. That way the user can basically forget about pipelining which is a good thing :)

in event loop assertion failing with multiple threads

This line is getting hit when MultiThreadedEventLoopGroup is configured with numThreads > 1 right here.

You can recreate this with the following steps:

1: clone swift-nio master
2: configure group to System.coreCount threads or just hard-code to 8
3: run NIOHTTP1Server executable
4: run curl localhost:8888

This doesn't get hit in release mode (obviously) and wrk seems to work fine, but I'm assuming that assert there is important for performance / stability reasons.

Project Icon Missing

Seriously. An Apple project w/o a Logo? Maybe some swallow crashing into a S/390 or something? Or a bug running around in an Xserve? Neo from Matrix? Something!

Add better support for handling signals.

The traditional UNIX signal model is not really ideal for programs like NIO, where signal handlers are basically incapable of doing any real work that affects the event loops. Signal handlers are not able to obtain locks or do basically anything that may require that other code on the thread runs, so in a NIO program a signal handler can basically do nothing of interest: it can't even call malloc.

This is a bad scene, particularly in Swift, where there is very little Swift code that you can confidently run from a signal handler (no malloc, remember!). Given that interacting with signals is a common thing to want to do for servers (e.g. SIGHUP for reloading config), we should build an abstraction that allows NIO-using libraries to handle signals more gracefully. Naturally this will need to be opt-in, but that's ok.

The easiest way to do this is to implement the anonymous self-pipe trick, the standard solution to the problem (TL;DR at program start you create an anonymous pipe that you select on in your event loop, and in a signal handler you write a byte to the pipe and return, allowing you to handle the signal in the loop instead). An alternative option would be to spawn a thread dedicated to signal handling, though that may be a bit excessive (and it requires masking away the signals on all the other threads which is a PITA).

Regardless, we should at some point sit down and provide this abstraction.

Flaky test for HTTP delayed upgrade.

One of the HTTP delayed upgrade tests occasionally fails with the following backtrace:

* thread #7, name = 'NIO-ELT-#0', stop reason = Fatal error: Can't remove last element from an empty collection
    frame #0: 0x00000001094ff100 libswiftCore.dylib`_swift_runtime_on_report
    frame #1: 0x0000000109542bb1 libswiftCore.dylib`_swift_stdlib_reportFatalError + 113
    frame #2: 0x0000000109273d66 libswiftCore.dylib`function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> () to @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> (@out ()), Argument Types : [@callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> ()]> of generic specialization <()> of Swift.StaticString.withUTF8Buffer<A>((Swift.UnsafeBufferPointer<Swift.UInt8>) -> A) -> A + 54
    frame #3: 0x00000001094e3df3 libswiftCore.dylib`partial apply forwarder for closure #2 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._fatalErrorMessage(Swift.StaticString, Swift.StaticString, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 99
    frame #4: 0x0000000109273d66 libswiftCore.dylib`function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> () to @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> (@out ()), Argument Types : [@callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> ()]> of generic specialization <()> of Swift.StaticString.withUTF8Buffer<A>((Swift.UnsafeBufferPointer<Swift.UInt8>) -> A) -> A + 54
    frame #5: 0x00000001093fbfb0 libswiftCore.dylib`function signature specialization <Arg[2] = Dead, Arg[3] = Dead> of Swift._fatalErrorMessage(Swift.StaticString, Swift.StaticString, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never + 96
    frame #6: 0x0000000109368d8e libswiftCore.dylib`merged (extension in Swift):Swift.RangeReplaceableCollection< where A: Swift.BidirectionalCollection>.removeLast() -> A.Element + 526
    frame #7: 0x0000000109270dd0 libswiftCore.dylib`(extension in Swift):Swift.RangeReplaceableCollection< where A: Swift.BidirectionalCollection>.removeLast() -> A.Element + 16
    frame #8: 0x000000010438016d NIOHTTP1Tests`implicit closure #4 in assertResponseIs(lines=0x000070000f838f80) at HTTPUpgradeTests.swift:91
    frame #9: 0x00000001043a04e7 NIOHTTP1Tests`partial apply for implicit closure #4 in assertResponseIs(response:expectedResponseLine:expectedResponseHeaders:) at HTTPUpgradeTests.swift:0
    frame #10: 0x000000010431a67a NIOHTTP1Tests`thunk for @callee_guaranteed () -> (@owned String, @error @owned Error) at HTTPHeadersTest.swift:0
    frame #11: 0x00000001043a054b NIOHTTP1Tests`thunk for @callee_guaranteed () -> (@owned String, @error @owned Error)partial apply at HTTPUpgradeTests.swift:0
    frame #12: 0x0000000109c484f2 libswiftXCTest.dylib`merged closure #1 () throws -> () in XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 162
    frame #13: 0x0000000109c550da libswiftXCTest.dylib`merged partial apply forwarder for closure #1 () throws -> () in XCTest.XCTAssertGreaterThan<A where A: Swift.Comparable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 42
    frame #14: 0x0000000109c55247 libswiftXCTest.dylib`partial apply forwarder for closure #1 () throws -> () in XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 23
    frame #15: 0x0000000109c54f1b libswiftXCTest.dylib`partial apply forwarder for closure #1 () -> () in XCTest._XCTRunThrowableBlock(() throws -> ()) -> XCTest._XCTThrowableBlockResult + 27
    frame #16: 0x0000000109c4794e libswiftXCTest.dylib`reabstraction thunk helper from @callee_guaranteed () -> () to @callee_unowned @convention(block) () -> () + 14
    frame #17: 0x0000000109c5577e libswiftXCTest.dylib`_XCTRunThrowableBlockBridge + 14
    frame #18: 0x0000000109c48deb libswiftXCTest.dylib`function signature specialization <Arg[0] = Owned To Guaranteed> of XCTest._XCTRunThrowableBlock(() throws -> ()) -> XCTest._XCTThrowableBlockResult + 203
    frame #19: 0x0000000109c4c575 libswiftXCTest.dylib`function signature specialization <Arg[0] = Owned To Guaranteed, Arg[1] = Owned To Guaranteed, Arg[2] = Owned To Guaranteed, Arg[3] = Exploded> of XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 325
    frame #20: 0x0000000109c485e9 libswiftXCTest.dylib`merged XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 41
    frame #21: 0x0000000109c47c51 libswiftXCTest.dylib`XCTest.XCTAssertEqual<A where A: Swift.Equatable>(@autoclosure () throws -> A, @autoclosure () throws -> A, @autoclosure () -> Swift.String, file: Swift.StaticString, line: Swift.UInt) -> () + 81
    frame #22: 0x000000010437f5cf NIOHTTP1Tests`assertResponseIs(response="", expectedResponseLine="HTTP/1.1 101 Switching Protocols", expectedResponseHeaders=3 values) at HTTPUpgradeTests.swift:91
  * frame #23: 0x0000000104399ab6 NIOHTTP1Tests`closure #2 in HTTPUpgradeTestCase.testDelayedUpgradeBehaviour(buffers=0 values, completePromise=NIO.EventLoopPromise<Swift.Void> @ 0x000070000f8390e0) at HTTPUpgradeTests.swift:658
    frame #24: 0x0000000104399b4d NIOHTTP1Tests`partial apply for closure #2 in HTTPUpgradeTestCase.testDelayedUpgradeBehaviour() at HTTPUpgradeTests.swift:0
    frame #25: 0x000000010434d270 NIOHTTP1Tests`closure #1 in ArrayAccumulationHandler.init(completion=0x0000000104399b40 NIOHTTP1Tests`partial apply forwarder for closure #2 (Swift.Array<NIO.ByteBuffer>) -> () in NIOHTTP1Tests.HTTPUpgradeTestCase.testDelayedUpgradeBehaviour() throws -> () at HTTPUpgradeTests.swift, self=0x000000010234efa0) at HTTPServerClientTest.swift:58
    frame #26: 0x000000010434d314 NIOHTTP1Tests`partial apply for closure #1 in ArrayAccumulationHandler.init(completion:) at HTTPServerClientTest.swift:0
    frame #27: 0x000000010434d34d NIOHTTP1Tests`thunk for @escaping @callee_guaranteed () -> () at HTTPServerClientTest.swift:0
    frame #28: 0x00007fff5a4fc0c9 libdispatch.dylib`_dispatch_client_callout + 8
    frame #29: 0x00007fff5a50ca2f libdispatch.dylib`_dispatch_block_invoke_direct + 294
    frame #30: 0x000000010434d91f NIOHTTP1Tests`ArrayAccumulationHandler.channelUnregistered(ctx=0x00000001027d2170, self=0x000000010234efa0) at HTTPServerClientTest.swift:67
    frame #31: 0x000000010434dc51 NIOHTTP1Tests`protocol witness for _ChannelInboundHandler.channelUnregistered(ctx:) in conformance ArrayAccumulationHandler<A> at HTTPServerClientTest.swift:0
    frame #32: 0x0000000104599b20 NIO`ChannelHandlerContext.invokeChannelUnregistered(self=0x00000001027d2170) at ChannelPipeline.swift:1028
    frame #33: 0x00000001045932de NIO`ChannelPipeline.fireChannelUnregistered0(self=0x0000000102608b30) at ChannelPipeline.swift:615
    frame #34: 0x000000010462ea30 NIO`BaseSocketChannel.close0(error=alreadyClosed, mode=all, promise=nil, self=0x00000001026084a0) at SocketChannel.swift:495
    frame #35: 0x00000001046376a4 NIO`SocketChannel.close0(error=alreadyClosed, mode=all, promise=nil, self=0x00000001026084a0) at SocketChannel.swift:929
    frame #36: 0x0000000104632047 NIO`protocol witness for ChannelCore.close0(error:mode:promise:) in conformance BaseSocketChannel<A> at SocketChannel.swift:0
    frame #37: 0x000000010459cbc9 NIO`HeadChannelHandler.close(ctx=0x0000000102608b80, mode=all, promise=nil, self=0x000000010279ec00) at ChannelPipeline.swift:707
    frame #38: 0x000000010459cf2f NIO`protocol witness for _ChannelOutboundHandler.close(ctx:mode:promise:) in conformance HeadChannelHandler at ChannelPipeline.swift:0
    frame #39: 0x0000000104596611 NIO`ChannelHandlerContext.invokeClose(mode=all, promise=nil, self=0x0000000102608b80) at ChannelPipeline.swift:1189
    frame #40: 0x00000001045966cb NIO`ChannelHandlerContext.invokeClose(mode=all, promise=nil, self=0x00000001027d2170) at ChannelPipeline.swift:1191
    frame #41: 0x00000001045943f2 NIO`ChannelPipeline.close0(mode=all, promise=nil, self=0x0000000102608b30) at ChannelPipeline.swift:541
    frame #42: 0x00000001045944fe NIO`closure #1 in ChannelPipeline.close(self=0x0000000102608b30, mode=all, promise=nil) at ChannelPipeline.swift:444
    frame #43: 0x000000010459b245 NIO`partial apply for closure #1 in ChannelPipeline.close(mode:promise:) at ChannelPipeline.swift:0
    frame #44: 0x0000000104561cfc NIO`thunk for @escaping @callee_guaranteed () -> () at Bootstrap.swift:0
    frame #45: 0x00000001045bba81 NIO`partial apply for thunk for @escaping @callee_guaranteed () -> () at EventLoop.swift:0
    frame #46: 0x00000001045bb827 NIO`thunk for @escaping @callee_guaranteed (@in ()) -> (@out ()) at EventLoop.swift:0
    frame #47: 0x00000001045bba21 NIO`thunk for @escaping @callee_guaranteed (@in ()) -> (@out ())partial apply at EventLoop.swift:0
    frame #48: 0x00000001045bb847 NIO`closure #2 in SelectableEventLoop.run(task=0x00000001045bba10 NIO`reabstraction thunk helper from @escaping @callee_guaranteed (@in ()) -> (@out ()) to @escaping @callee_guaranteed () -> ()partial apply forwarder with unmangled suffix ".57" at EventLoop.swift) at EventLoop.swift:527
    frame #49: 0x00000001045bb8a1 NIO`partial apply for closure #2 in SelectableEventLoop.run() at EventLoop.swift:0
    frame #50: 0x0000000104558da6 NIO`thunk for @callee_guaranteed () -> (@error @owned Error) at BlockingIOThreadPool.swift:0
    frame #51: 0x00000001045bb90b NIO`thunk for @callee_guaranteed () -> (@error @owned Error)partial apply at EventLoop.swift:0
    frame #52: 0x00000001045b5eb2 NIO`closure #1 in withAutoReleasePool<A>(execute=0x00000001045bb8f0 NIO`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @callee_guaranteed () -> (@out (), @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".50" at EventLoop.swift) at EventLoop.swift:261
    frame #53: 0x00000001045c33f6 NIO`partial apply for closure #1 in withAutoReleasePool<A>(_:) at EventLoop.swift:0
    frame #54: 0x0000000109bb687f libswiftObjectiveC.dylib`ObjectiveC.autoreleasepool<A>(invoking: () throws -> A) throws -> A + 47
    frame #55: 0x00000001045b5e28 NIO`withAutoReleasePool<A>(execute=0x00000001045bb8f0 NIO`reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @callee_guaranteed () -> (@out (), @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".50" at EventLoop.swift) at EventLoop.swift:260
    frame #56: 0x00000001045bada2 NIO`SelectableEventLoop.run(self=0x00000001027d10c0) at EventLoop.swift:526
    frame #57: 0x00000001045bdedf NIO`closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(t=(pthread = 0x000070000f83b000), initializer=0x00000001045c2e30 NIO`partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed (@in NIO.Thread) -> (@out ()) to @escaping @callee_guaranteed (@owned NIO.Thread) -> () at EventLoop.swift, lock=(mutex = 0x0000000109d38ff0), _loop=0x00000001027d10c0, loopUpAndRunningGroup=0x0000000109d38e50) at EventLoop.swift:666
    frame #58: 0x00000001045c2f0d NIO`partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:initializer:) at EventLoop.swift:0
    frame #59: 0x00000001045be48f NIO`thunk for @escaping @callee_guaranteed (@owned Thread) -> () at EventLoop.swift:0
    frame #60: 0x000000010464cca1 NIO`partial apply for thunk for @escaping @callee_guaranteed (@owned Thread) -> () at Thread.swift:0
    frame #61: 0x000000010464bcf8 NIO`closure #1 in static Thread.spawnAndRun(p=(_rawValue = 0x0000000109d37040 -> 0x00000001095b86a0 libswiftCore.dylib`InitialAllocationPool + 13968)) at Thread.swift:105
    frame #62: 0x000000010464bd79 NIO`@objc closure #1 in static Thread.spawnAndRun(name:body:) at Thread.swift:0
    frame #63: 0x00007fff5a851642 libsystem_pthread.dylib`_pthread_body + 126
    frame #64: 0x00007fff5a8515c4 libsystem_pthread.dylib`_pthread_start + 61
    frame #65: 0x00007fff5a851229 libsystem_pthread.dylib`thread_start + 13

We should investigate why this is happening, but it seems to just be a problem in the test code itself: probably a race.

No non-blocking file-open operation

Expected behavior

The way to open files seems to be:

NIO.FileHandle.init(path:)

which is a blocking operation. That seems no good.

Actual behavior

A non-blocking way to open a file. Maybe this belongs into NonBlockingFileIO.

SwiftNIO version/commit hash

The HEAD when I pressed "Submit".

CI: Docker image for testing throws quite a few errors, no consistent base image

Expected behavior

  • The docker build should run w/o errors. That doesn't increase trust in a test environment.
  • Also for a test environment, there should be an image in the first place. Otherwise you can guarantee reproducibility. (and it makes the setup slow as a minor inconvenience, but that somewhat matters for contributors who actually just want to contribute a single line fix). The current image pulls the latest versions of packages, which may very well get out of sync.

Actual behavior

Setting up the local image from the Dockerfile produces quite a few errors, looks as if mostly related to not having an interactive terminal. For example:

 ---> 53ee6818e23b
Step 4/39 : RUN dpkg-reconfigure locales
 ---> Running in a4728f63b96c
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
Generating locales...
  en_US.ISO-8859-1... up-to-date
 Removing any system startup links for /etc/init.d/rsync ...
update-rc.d: warning: default stop runlevel arguments (0 1 6) do not match rsync Default-Stop values (none)
 Adding system startup for /etc/init.d/rsync ...
   /etc/rc0.d/K20rsync -> ../init.d/rsync
...
invoke-rc.d: policy-rc.d denied execution of restart.

There are quite a lot of those errors, maybe around 20-30 (showing up red in the macOS terminal).

Steps to reproduce

  1. in a clean docker environment, call docker-compose -f docker/docker-compose.yaml up test

If possible, minimal yet complete reproducer code (or URL to code)

docker-compose -f docker/docker-compose.yaml up test

SwiftNIO version/commit hash

This is on the helje5:bugs/fix-issue-110 fork branch, hash b2920b6, but that shouldn't matter.

Swift & OS version (output of swift --version && uname -a)

Apple Swift version 4.0.3 (swiftlang-900.0.74.1 clang-900.0.39.2)
Target: x86_64-apple-macosx10.9
Darwin ZeeMBP 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

Docker: Version 17.12.0-ce-mac55 (23011), Channel: stable, d62ef8d1b0

clang 3.6 not supported

It seems that clang 3.8 or later is required for use with Swift NIO.

Environment

-> /swift/swift-4.1-DEVELOPMENT-SNAPSHOT-2018-03-12-a-ubuntu16.04/usr/bin/swift --version && clang --version
Swift version 4.1-dev (LLVM 1c8b50929b, Clang 420ae40df6, Swift ec821efb01)
Target: x86_64-unknown-linux-gnu
Ubuntu clang version 3.6.2-3ubuntu2 (tags/RELEASE_362/final) (based on LLVM 3.6.2)
Target: x86_64-pc-linux-gnu
Thread model: posix

-> /swift/4.1.0-beta/usr/bin/swift --version && clang --version
Swift version 4.1-dev (LLVM 4d75b446be, Clang db92199adc, Swift 4cb7e7f233)
Target: x86_64-unknown-linux-gnu
Ubuntu clang version 3.6.2-3ubuntu2 (tags/RELEASE_362/final) (based on LLVM 3.6.2)
Target: x86_64-pc-linux-gnu
Thread model: posix
-> lsb_release -a
No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:    16.04
Codename:    xenial

Error

Compile CNIOAtomics src/c-atomics.c
Compile CNIOLinux shim.c
Compile CNIODarwin shim.c
Compile CNIOZlib empty.c
Compile CNIOHTTPParser c_nio_http_parser.c
Compile Swift Module 'Debugging' (4 sources)
In file included from /vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/c-atomics.c:21:
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:20:37: error: expected ';' after top level declarator
struct catmc_atomic__Bool * _Nonnull catmc_atomic__Bool_create(bool value);
                                    ^
                                    ;
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:21:70: error: expected ')'
void catmc_atomic__Bool_destroy(struct catmc_atomic__Bool * _Nonnull atomic);
                                                                     ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:21:32: note: to match this '('
void catmc_atomic__Bool_destroy(struct catmc_atomic__Bool * _Nonnull atomic);
                               ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:22:83: error: expected ')'
bool catmc_atomic__Bool_compare_and_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool expected, bool desired);
                                                                                  ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:22:45: note: to match this '('
bool catmc_atomic__Bool_compare_and_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool expected, bool desired);
                                            ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:23:66: error: expected ')'
bool catmc_atomic__Bool_add(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                                                                 ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:23:28: note: to match this '('
bool catmc_atomic__Bool_add(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                           ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:24:66: error: expected ')'
bool catmc_atomic__Bool_sub(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                                                                 ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:24:28: note: to match this '('
bool catmc_atomic__Bool_sub(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                           ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:25:71: error: expected ')'
bool catmc_atomic__Bool_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                                                                      ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:25:33: note: to match this '('
bool catmc_atomic__Bool_exchange(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                                ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:26:67: error: expected ')'
bool catmc_atomic__Bool_load(struct catmc_atomic__Bool * _Nonnull atomic);
                                                                  ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:26:29: note: to match this '('
bool catmc_atomic__Bool_load(struct catmc_atomic__Bool * _Nonnull atomic);
                            ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:27:68: error: expected ')'
void catmc_atomic__Bool_store(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                                                                   ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:27:30: note: to match this '('
void catmc_atomic__Bool_store(struct catmc_atomic__Bool * _Nonnull atomic, bool value);
                             ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:29:28: error: redefinition of '_Nonnull' with a different type: 'struct catmc_atomic_char *' vs 'struct catmc_atomic__Bool *'
struct catmc_atomic_char * _Nonnull catmc_atomic_char_create(char value);
                           ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:20:29: note: previous definition is here
struct catmc_atomic__Bool * _Nonnull catmc_atomic__Bool_create(bool value);
                            ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:29:36: error: expected ';' after top level declarator
struct catmc_atomic_char * _Nonnull catmc_atomic_char_create(char value);
                                   ^
                                   ;
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:30:68: error: expected ')'
void catmc_atomic_char_destroy(struct catmc_atomic_char * _Nonnull atomic);
                                                                   ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:30:31: note: to match this '('
void catmc_atomic_char_destroy(struct catmc_atomic_char * _Nonnull atomic);
                              ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:31:81: error: expected ')'
bool catmc_atomic_char_compare_and_exchange(struct catmc_atomic_char * _Nonnull atomic, char expected, char desired);
                                                                                ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:31:44: note: to match this '('
bool catmc_atomic_char_compare_and_exchange(struct catmc_atomic_char * _Nonnull atomic, char expected, char desired);
                                           ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:32:64: error: expected ')'
char catmc_atomic_char_add(struct catmc_atomic_char * _Nonnull atomic, char value);
                                                               ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:32:27: note: to match this '('
char catmc_atomic_char_add(struct catmc_atomic_char * _Nonnull atomic, char value);
                          ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:33:64: error: expected ')'
char catmc_atomic_char_sub(struct catmc_atomic_char * _Nonnull atomic, char value);
                                                               ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:33:27: note: to match this '('
char catmc_atomic_char_sub(struct catmc_atomic_char * _Nonnull atomic, char value);
                          ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:34:69: error: expected ')'
char catmc_atomic_char_exchange(struct catmc_atomic_char * _Nonnull atomic, char value);
                                                                    ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:34:32: note: to match this '('
char catmc_atomic_char_exchange(struct catmc_atomic_char * _Nonnull atomic, char value);
                               ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:35:65: error: expected ')'
char catmc_atomic_char_load(struct catmc_atomic_char * _Nonnull atomic);
                                                                ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:35:28: note: to match this '('
char catmc_atomic_char_load(struct catmc_atomic_char * _Nonnull atomic);
                           ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:36:66: error: expected ')'
void catmc_atomic_char_store(struct catmc_atomic_char * _Nonnull atomic, char value);
                                                                 ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:36:29: note: to match this '('
void catmc_atomic_char_store(struct catmc_atomic_char * _Nonnull atomic, char value);
                            ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:38:29: error: redefinition of '_Nonnull' with a different type: 'struct catmc_atomic_short *' vs 'struct catmc_atomic__Bool *'
struct catmc_atomic_short * _Nonnull catmc_atomic_short_create(short value);
                            ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:20:29: note: previous definition is here
struct catmc_atomic__Bool * _Nonnull catmc_atomic__Bool_create(bool value);
                            ^
/vapor/hi-nodes-production/code/.build/checkouts/swift-nio.git-8876730210459286346/Sources/CNIOAtomics/src/../include/c-atomics.h:38:37: error: expected ';' after top level declarator
struct catmc_atomic_short * _Nonnull catmc_atomic_short_create(short value);
                                    ^
                                    ;
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: terminated(1): /swift/4.1.0-beta/usr/bin/swift-build-tool -f /vapor/hi-nodes-production/code/.build/release.yaml main output:


Fetching https://github.com/vapor/console.git
Fetching https://github.com/vapor/database-kit.git
Fetching https://github.com/vapor/routing.git
Fetching https://github.com/apple/swift-nio.git
Fetching https://github.com/vapor/service.git
Fetching https://github.com/vapor/validation.git
Fetching https://github.com/vapor/crypto.git
Fetching https://github.com/apple/swift-nio-zlib-support.git
Fetching https://github.com/vapor/template-kit.git
Fetching https://github.com/vapor/vapor.git
Fetching https://github.com/vapor/core.git
Fetching https://github.com/vapor/engine.git
Fetching https://github.com/vapor/copenssl.git
Cloning https://github.com/apple/swift-nio-zlib-support.git
Resolving https://github.com/apple/swift-nio-zlib-support.git at 1.0.0
Cloning https://github.com/vapor/vapor.git
Resolving https://github.com/vapor/vapor.git at nio
Cloning https://github.com/vapor/core.git
Resolving https://github.com/vapor/core.git at nio
Cloning https://github.com/vapor/copenssl.git
Resolving https://github.com/vapor/copenssl.git at 1.0.0-rc.1
Cloning https://github.com/vapor/console.git
Resolving https://github.com/vapor/console.git at nio
Cloning https://github.com/vapor/database-kit.git
Resolving https://github.com/vapor/database-kit.git at nio
Cloning https://github.com/vapor/routing.git
Resolving https://github.com/vapor/routing.git at nio
Cloning https://github.com/apple/swift-nio.git
Resolving https://github.com/apple/swift-nio.git at 1.1.0
Cloning https://github.com/vapor/service.git
Resolving https://github.com/vapor/service.git at nio
Cloning https://github.com/vapor/validation.git
Resolving https://github.com/vapor/validation.git at nio
Cloning https://github.com/vapor/crypto.git
Resolving https://github.com/vapor/crypto.git at nio
Cloning https://github.com/vapor/template-kit.git
Resolving https://github.com/vapor/template-kit.git at nio
Cloning https://github.com/vapor/engine.git
Resolving https://github.com/vapor/engine.git at nio

Error: deploy failed.

Is ByteBuffer thread-safe?

Sorry, if this question is dumb, because the whole purpose of the object may be to allow exactly this, but the documentation does not specifically clarify this.

Is it safe to write from one thread and read from another?

Build an API for a writable view or slice.

Discussions made it clear that the CoW API of ByteBuffer means that slice cannot be used to provide a writable window into a larger ByteBuffer. This is because any attempt to write into the ByteBuffer produced by slice will cause a CoW, leading to the write happening in a new buffer.

We should add a new method (or family of methods), tentatively called withWritableBufferView, that will provide a temporary window into the ByteBuffer's underlying storage that escapes the CoW implementation. This can probably only be done by wrapping an UnsafeMutablePointer to the inner buffer, which is why it seems sensible to make this a with... method: it ensures that the lifetime of the writable buffer view is strictly controlled.

The BackPressure handler should be dropped

The included BackPressureHandler only applies to a non-existing use case and gives a false sense of back-pressure-security. It is DoS-dangerous and should rather be dropped completely, or at least renamed to something like PassThroughBackPressureHandler. It should not be advertised to be used, only with very specific instructions when it actually applies (like echo ;-) ).

Rational

The idea of a back pressure is that you don't read more data than what the stack can process. I.e. that you stop reading data you can't process anyways at the socket level. This is particularly important with async services because they don't block the reading side during processing (BP is automatic/implicit w/ synchronous services).

Status

The included BackPressureHandler ties readability to the writability of the socket, which is non-sensical except for the super rare use case of a pure "transforming" aka "echo" service. Aka "don't read more unless I can write".

What makes it so bad is that the current setup suggests that you have proper BP handling, when in fact you don't have any at all.

Actual Backpressure

In async services BP is a must have all over. For the common case the socket will stay writable all the time, which makes the BP included useless. Only if the processing completes, a service will issue a 200 OK. Maybe sometimes it will issue a 100 Continue to keep the socket alive.

Example

Lets imagine you enable iCloud on your device for the first time. This will result in an upload of your vCards and iCalendars in bulk.
Overly simplified it will be a POST-2000-vCards, and a POST-5000-iCalendar-Events.

Ignoring watermarks and such to facilitate batch processing (which NIO doesn't have), imagine this as:

  • inbound 1 vCard
  • db connection INSERT 1 vCard
  • inbound 2 vCard
  • db connection INSERT 2 vCard

Nothing is every written to the socket-out (except maybe the keep-alive 100-continues). Which means the included BP handler will never kick in.

In the current setup a service would accumulate all 2000 vCards in memory (schedule them to be inserted), w/o considering database insert performance (which will lag a lot).

Other Examples

Pretty much everything which involves outgoing I/O. Say fetching weather info or stock prices. In other words, which makes async stuff async ;-)

Desired Behaviour

The BP must not need to be tied to the client-socket writability, but to the database-socket writability. I.e. can the DB keep up w/ inserting the records.


I hope this makes sense and isn't too confusing (and god I hope I'm not talking non-sense and missed something which actually makes it work as desired ;-) ).

FWIW I'm not advocating that NIO does BP, it should probably be done at the layer above. My suggestion is that "The BackPressure handler should be dropped" as in the title. It is pretty much useless and very much misleading.

publicize `NIOFoundationCompat`

Is there a reason this package was not added to the products array in the package manifest?

If that's just an oversight, I'm happy to submit a PR. :)

Correctly delete `pthread_key`s

Expected behavior

You'd expect all memory allocated when creating a ThreadSpecificVariable to be deallocated when it goes out of scope.

Actual behavior

The pthread_key might actually never be deallocated as we never call pthread_key_delete.

SwiftNIO version/commit hash

all versions up to and including 1.1.0

Swift & OS version (output of swift --version && uname -a)

doesn't matter

Why this is hard

If we call pthread_key_delete in ThreadSpecificVariable's deinit (after turning it into a class), then the destructor passed to pthread_key_create will not be called anymore. That means we'd leak values that have been set to this ThreadSpecificVariable which is unacceptable.
That means we can only call pthread_key_delete if both the following conditions are met:

  • the ThreadSpecificVariable has been deallocated
  • all threads that had a value set to the ThreadSpecificVariable have exited (and therefore run the destructor passed to pthread_key_create)

The only way I could see that we achieve this is to store a Box<(ThreadSpecficVariable, T)>? in the pthread_setspecific because that way the ThreadSpecificVariable will be alive as long as all threads that have a value set.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.