handleToFd leaks memory about unix HOT 14 CLOSED

I can't reproduce this. I see variable but bounded virtual memory size for this program, never exceeding about 60M, with resident size around half that.

from unix.

What version of unix are you using and what version of GHC? I can
reproduce it quite reliably with the included program, on some setups it
consumes several GBs of virtual memory in a few seconds.

On Tue, Jul 29, 2014 at 9:24 PM, rwbarton [email protected] wrote:

I can't reproduce this. I see variable but bounded virtual memory size for
this program, never exceeding about 60M, with resident size around half
that.

—
Reply to this email directly or view it on GitHub
#4 (comment).

from unix.

@pudlak How are you compiling the program? Are you passing in any RTS options?

from unix.

@snoyberg I don't pass any optimization or RTS options, but anyway it seems it doesn't really matter. Here is a cabal setup that produces such an executable:

name:                leak
version:             0.1.0.0
license:             PublicDomain
author:              Petr Pudlak
build-type:          Simple
cabal-version:       >=1.10

executable leak
  build-depends:       base >=4.6 && <4.7, unix ==2.7.0.1
  default-language:    Haskell2010
  main-is:             Leak.hs

I always compile and run it inside a schroot, IDK if this could somehow contribute to the problem. I tested it in both Wheezy (GHC 7.4.1, unix-2.5.0.1) and Squeeze schroot (GHC 7.6.3 and unix-2.[6,7].0.1), the problem manifests in all these setups.

from unix.

I'm still unable to repro. Can you try without schroot and see if it still occurs?

from unix.

Can you reproduce the problem with a Dockerfile or vagrant script? Given the trouble others are having reproducing this, if we can get a repeatable build setup that illustrates the bug via VM/container that would be helpful here.

from unix.

@gregorycollins I'll try to do that. @snoyberg So far I'm unable to reproduce it outside schroot as well.

from unix.

It can be reproduced in a docker image my coworker (@nh2) created: nh2docker/ganeti-ubuntu1404-code-syshs
The image already contains GHC, just compile the sample code from above and run. It eats around 100MB RAM per second.

from unix.

I can reproduce this using the docker image.

from unix.

@nomeata any idea on how to fix this?

from unix.

This seems to be some issue with broken locales related to GHC Trac #7695 and #10298. I can reproduce it on an ordinary system after export LC_ALL=badlc. I'd guess the allocations are happening in iconv.

from unix.

Nice find. It doesn't leak very quickly, but this definitely triggers the issue on my machine.

from unix.

OK, the cause is rather simple. Compare the eventual implementation of hClose from base

hClose_handle_ :: Handle__ -> IO (Handle__, Maybe SomeException)
hClose_handle_ h_@Handle__{..} = do

    -- close the file descriptor, but not when this is the read
    -- side of a duplex handle.
    -- If an exception is raised by the close(), we want to continue
    -- to close the handle and release the lock if it has one, then
    -- we return the exception to the caller of hClose_help which can
    -- raise it if necessary.
    maybe_exception <-
      case haOtherSide of
        Nothing -> trymaybe $ IODevice.close haDevice
        Just _  -> return Nothing

    -- free the spare buffers
    writeIORef haBuffers BufferListNil
    writeIORef haCharBuffer noCharBuffer
    writeIORef haByteBuffer noByteBuffer

    -- release our encoder/decoder
    closeTextCodecs h_

    -- we must set the fd to -1, because the finalizer is going
    -- to run eventually and try to close/unlock it.
    -- ToDo: necessary?  the handle will be marked ClosedHandle
    -- XXX GHC won't let us use record update here, hence wildcards
    return (Handle__{ haType = ClosedHandle, .. }, maybe_exception)

to that of handleToFd from unix

handleToFd' :: Handle -> Handle__ -> IO (Handle__, Fd)
handleToFd' h h_@Handle__{haType=_,..} = do
  case cast haDevice of
    Nothing -> ioError (ioeSetErrorString (mkIOError IllegalOperation
                                           "handleToFd" (Just h) Nothing)
                        "handle is not a file descriptor")
    Just fd -> do
     -- converting a Handle into an Fd effectively means
     -- letting go of the Handle; it is put into a closed
     -- state as a result.
     flushWriteBuffer h_
     FD.release fd
     return (Handle__{haType=ClosedHandle,..}, Fd (FD.fdFD fd))

flushWriteBuffer is just

flushWriteBuffer h_@Handle__{..} = do
  buf <- readIORef haByteBuffer
  when (isWriteBuffer buf) $ flushByteWriteBuffer h_

so clearly handleToFd' is rather delinquent in freeing resources, particularly the text codecs.

The reason the (large) leak does not normally appear is just that GHC implements UTF-8 encoding itself, without using iconv. Any locale for which GHC uses iconv produces the same behavior, not just non-existent locales.

It looks like this should be fixable by just copying the relevant parts of hClose_handle_ into handleToFd', though the resulting cross-package code duplication is unfortunate.

from unix.

Tried to reproduce it in 2022... the only GHC I could reproduce it with is 8.0.2. Next unix release won't support 8.0.2 or earlier, so it appears this issue is resolved.

from unix.