Skip to content

Unable to start slapdtest on MacOS #422

Open
Open
Unable to start slapdtest on MacOS#422

Description

@dwt
dwt
opened on Aug 18, 2021

Issue description:

Cannot start slapdtest.SlapObject() on MacOS 11.5.1

Steps to reproduce:

import pytest
from pyexpect import expect
import slapdtest

@pytest.fixture(scope='session')
def slapd():
    # homebrew installs openldap as keg-only, thus the path needs to be added
    import subprocess
    import os
    
    # prefix = subprocess.run(['brew', '--prefix'], capture_output=True).stdout.strip().decode()
    # openldap_path = f'{prefix}/opt/openldap'
    # bin_path = f'{openldap_path}/bin'
    # sbin_path = f'{openldap_path}/sbin'
    # libexec_path = f'{openldap_path}/libexec'
    # fixed_path = f'{bin_path}:{sbin_path}:{libexec_path}:' + os.environ['PATH']
    fixed_path = '/usr/libexec/:' + os.environ['PATH']
    slapdtest.SlapdObject.BIN_PATH = fixed_path
    slapdtest.SlapdObject.SBIN_PATH = fixed_path
    
    import logging
    slapdtest.SlapdObject._log.setLevel(logging.DEBUG)
    
    with slapdtest.SlapdObject() as slapd:
        yield slapd
    
def test_smoke(slapd):
    pass

Both the native version provided by apple, as well as the current version from homebrew fail to start - albeit with different reasons.

The Apple provided

% /usr/libexec/slapd -V                                                     :(
@(#) $OpenLDAP: slapd 2.4.28 (Jun 17 2021 16:09:47) $
	[email protected]:/System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Binaries/OpenLDAP/install/TempContent/Objects/servers/slapd

dies with this error

611d131f /path/to/testdir/slapd.conf: line 12: <database> failed init (mdb)
slaptest: bad configuration file!

That a would be this line database mdb.

The Homebrew version gets slightly further:

~/Library/Homebrew/opt/openldap/libexec/slapd -V                          :(
@(#) $OpenLDAP: slapd 2.5.6 (Aug 18 2021 13:38:48) $
	openldap

this dies with the error

611d1725.219415a8 0x104d79e00 backend_startup_one: starting "cn=config"
611d1725.21951778 0x104d79e00 config_setup_ldif: expected directory /path/to/testdir/python-ldap-test-60577 to be empty!
611d1725.21957d08 0x104d79e00 slapd destroy: freeing system resources.
611d1725.219f7f88 0x104d79e00 slapd stopped.
611d1725.219fe130 0x104d79e00 connections_destroy: nothing to destroy.

I'm not quite sure what to make of this - is there documentation on the supported versions of openldap somewhere? Am I missing a setting I should have applied?

Operating system: Darwin nathan.fritz.box 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64
Python version: Python 3.9.6
python-ldap version: 3.3.1

Activity

  1. dwt commented on Aug 23, 2021

    @dwt
    dwt
    on Aug 23, 2021
    Author

    Any ideas how to best continue with this? As a workaround I've tried execution the unit tests in a debian-docker container with

    (venv) auth@803d93942a80:~/src$ slapd -V
@(#) $OpenLDAP: slapd  (Feb 14 2021 18:32:34) $
	Debian OpenLDAP Maintainers <[email protected]>
(venv) auth@803d93942a80:~/src$ apt info slapd
Package: slapd
Version: 2.4.47+dfsg-3+deb10u6

    This seems to work - insofar as I'm now able to start the slap daemon.

    I would like to run the test suite natively too if possible as it helps me bring additional debugging-tools to play that are harder to use inside the docker container. Any pointers?

  2. vashirov commented on Aug 23, 2021

    @vashirov
    vashirov
    on Aug 23, 2021

    Regarding the first error:

    611d131f /path/to/testdir/slapd.conf: line 12: <database> failed init (mdb)

    Build of OpenLDAP that comes with macOS doesn't have mdb backend (you can verify it with running /usr/libexec/slapd -VVV).

    Looking at your last error:

    611d1725.21951778 0x104d79e00 config_setup_ldif: expected directory /path/to/testdir/python-ldap-test-60577 to be empty!

    It's a result of the fix for ITS#9016
    This is how slapd was started in python-ldap-3.3.1:

    python-ldap/Lib/slapdtest/_slapdtest.py

    Lines 419 to 422 in dae0c51

    self.PATH_SLAPD,
    '-f', self._slapd_conf,
    '-F', self.testrundir,
    '-h', ' '.join(urls),

    Since testrundir is not empty and has files, slapd complains and fails to start.

    In #382 the configuration was changed from using slapd.conf to use slapd.d. Unfortunately, it doesn't work with the version of OpenLDAP that comes from homebrew, because it is built without modules support, and the test fails with:

    E           RuntimeError: ['/usr/local/opt/openldap/sbin/slapadd', '-F', '/Users/user/src/test/python-ldap-test-51436/slapd.d', '-n0'] process failed:
E           b''
E           b'slapadd: dn="cn=module,cn=config" (line=14): (65) unrecognized objectClass \'olcModuleList\'\n'

    So, you can either patch python-ldap-3.3.1 that comes from pypi with something like this:

    diff --git a/Lib/slapdtest/_slapdtest.py b/Lib/slapdtest/_slapdtest.py
index afd5304..a7a5c63 100644
--- a/Lib/slapdtest/_slapdtest.py
+++ b/Lib/slapdtest/_slapdtest.py
@@ -190,6 +190,7 @@ class SlapdObject(object):
     local_host = LOCALHOST
     testrunsubdirs = (
         'schema',
+        'slapd.d',
     )
     openldap_schema_files = (
         'core.schema',
@@ -220,6 +221,7 @@ class SlapdObject(object):
         self.server_id = self._port % 4096
         self.testrundir = os.path.join(self.TMPDIR, 'python-ldap-test-%d' % self._port)
         self._schema_prefix = os.path.join(self.testrundir, 'schema')
+        self._slapd_d = os.path.join(self.testrundir, 'slapd.d')
         self._slapd_conf = os.path.join(self.testrundir, 'slapd.conf')
         self._db_directory = os.path.join(self.testrundir, "openldap-data")
         self.ldap_uri = "ldap://%s:%d/" % (self.local_host, self._port)
@@ -418,7 +420,7 @@ class SlapdObject(object):
         slapd_args = [
             self.PATH_SLAPD,
             '-f', self._slapd_conf,
-            '-F', self.testrundir,
+            '-F', self._slapd_d,
             '-h', ' '.join(urls),
         ]
         if self._log.isEnabledFor(logging.DEBUG):

    or add --enable-modules in openldap.rb formulae and rebuild it from source.

  3. dwt commented on Aug 24, 2021

    @dwt
    dwt
    on Aug 24, 2021
    Author

    Thanks for the detailed reply, will try asap.

  4. hoefling commented on Oct 28, 2021

    @hoefling
    hoefling
    on Oct 28, 2021

    For reference: SlapdObject patching at runtime, no code modifications in python-ldap

    @contextlib.contextmanager
def __patch_slapd_start(server: SlapdObject):
    start_routine_orig = SlapdObject._start_slapd

    def _start_slapd_patched(self: SlapdObject) -> None:
        with patch.object(self, 'testrundir', self._slapd_d) as m:
            start_routine_orig(self)

    with patch.object(SlapdObject, '_start_slapd', _start_slapd_patched):
        yield


@pytest.fixture
def ldap_server():
    server = SlapdObject()
    server.database = 'ldif'
    server.testrunsubdirs += ('slapd.d', )
    server._slapd_d = str(Path(server.testrundir, 'slapd.d'))

    with __patch_slapd_start(server):
        server.start()

    yield server
    server.stop()

  5. jirutka commented on Nov 2, 2021

    @jirutka
    jirutka
    on Nov 2, 2021
    Contributor

    I’ve encountered the same problem on Alpine Linux. We do provide MDB backend, but it’s built as a dynamic module.

  6. jirutka commented on Nov 2, 2021

    @jirutka
    jirutka
    on Nov 2, 2021
    Contributor

    I’m trying vashirov’s patch from the comment above (#422 (comment)). It fixed most of the test failures, but not all. There are still expected directory … to be empty! errors from tests that restart LDAP (e.g. Tests.t_ldapobject.Test01_ReconnectLDAPObject). Okay, so I’m adding cleanup of _slapd_d directory, it fixes another bunch of failures, but again, not all of them.

    I ended up removing -F option from _start_slapd. Now all tests pass.

  7. mbanck commented on Apr 24, 2022

    @mbanck
    mbanck
    on Apr 24, 2022

    This is also a problem on Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008355

  8. mistotebe commented on Apr 26, 2022

    @mistotebe
    mistotebe
    on Apr 26, 2022
    Contributor

    @vashirov could you post your patch as a PR? Thanks

  9. mistotebe
    added this to the 3.4.1 milestone on Apr 26, 2022

  10. mistotebe commented on May 10, 2022

    @mistotebe
    mistotebe
    on May 10, 2022
    Contributor

    @vashirov could you post your patch as a PR? Thanks

    @vashirov or I can create it myself crediting you as author if you consent

  11. vashirov commented on May 24, 2022

    @vashirov
    vashirov
    on May 24, 2022

    @mistotebe, my patch was for 3.3.x. It is not required for 3.4.x that has changes from #382.
    Test from the issue description runs fine with python-ldap-3.4.0 on macOS 12.4 with openldap-2.6.2 from homebrew.
    It doesn't work with slapd that is shipped with macOS, because it's quite old (2.4.28) and it's not built with mdb support, but slapdtest uses it in the template:

    python-ldap/Lib/slapdtest/_slapdtest.py

    Line 184 in 7f30c47

    database = 'mdb'

    So I'm not sure if anything else should be done here.

  12. mistotebe commented on May 24, 2022

    @mistotebe
    mistotebe
    on May 24, 2022
    Contributor

    Oh, I see I missed that because superficially, your patch seemed it should still apply and definitely looked much cleaner than the current version... Thanks for the response @vashirov!

    At least that's one less thing to worry about and we can unblock 3.4.1. Still, not sure what to do about defaults on Mac OS except everyone passing database='hdb'/'ldif' to the SlapdObject constructor but we'll have to figure that out later.

  13. mistotebe
    removed this from the 3.4.1 milestone on May 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions