How does System.DirectoryServices know how to format an attribute value?

I use System.DirectoryServices all the time in my PowerShell scripts (I have a lot of client machines which don't have RSAT installed so I use System.DirectoryServices.DirectorySearcher instead) and for some reason it never occurred to me to ask this question. When I query a directory (Active Directory, OpenLDAP, or something else) how exactly does .NET know that the cn attribute is a string or that proxyAddresses is an array of strings? Since it just works I never thought to look into it, however as is almost always the case I was forced to ask and answer the question when a number of my scripts (and some .NET applications) stopped working properly when querying a non-AD directory.

The problem was simply this. When querying using S.DS.DS (System.DirectoryServices.DirectorySearcher) almost (but not all) results were coming back as an array of bytes instead of their proper types (strings, integers, booleans, etc..). Some attributes were however working.

Without going into too much detail sd is a PowerShell function that performs S.DS.DS searches against a variety of directories (AD and otherwise). Normally the output would be strings (or an array of strings), but instead as you can see its returning byte arrays. I had to remove the private stuff, but you can see from this that some attributes were properly being rendered, but most were not. This is not normal.

I fired up WireShark to take a look at was going on and the path to S.DS.DS enlightenment began. What I noticed was that while capturing LDAP queries with .Net it was doing something that was unexpected. It would go something like this:

1. Bind to the directory server (bindRequest, bindResult, searchRequest for RootDSE, searchResEntry, and finally searchResDone).
2. Perform the actual search (searchRequest, searchResEntry, and then searchResDone).
3. But this is what surprised me, under the hood it then does another query looking for (objectClass=subSchema) and asking for the modifyTimeStamp.

What was fascinating about #2 is that in the search results the requested attributes were NOT coming down as byte arrays, but as their proper string types. So it was S.DS.DS that was not rendering it right. And #3 was kind of interesting because I wasn't expecting it to do any sort of schema queries. I didn't know that this was a thing, but as it turns out some LDAP clients do this.

We'll get back to #2 in a moment, but what was happening when it was querying for the modifyTimeStamp of the subSchema was part of S.DS.DS's schema caching mechanism. What is going on here is that when S.DS.DS executes this query it compares the modifyTimeStamp with the Time value in the registry (if it exists) at HKCU\SOFTWARE\Microsoft\ADs\Providers\LDAP\schema DN. For example:

In this case it's the Active directory schema and you can see that it gives a file path and the Time. The value in Time is exactly the time that comes down from the directory server when the subSchema was queries. If the lastModifyTime from the schema is more recent (or there isn't an entry in the registry) S.DS.DS creates a new cache file and updates the registry. With WireShark I saw this in action where after it queried the directory servers subSchema it then executed another query asking for all objects in the schema and return their attributeTypes, objectClasses, and ditContentRules (this is AD specific, but it asks for it anyways).

If you look at the schema cache file it's pretty interesting stuff (for me anyways):

This actually answered a question that I never thought to ask. S.DS.DS queries the schema and if a newer one is found it creates a cache file which it loads into memory so that it knows how to render data that comes down from the directory server (it does this from any kind of LDAP server).

Now that we understand how this works we can get back to #2...our problem. What was interesting is that the registry key\cache wasn't being generated for the problematic directory server. I could point S.DS.DS at any other directory infrastructure we have (AD or otherwise) and it was using\generating cache files, but this one system was problematic. Going back to WireShark and comparing known good behavior with bad it was clear that in the problem interaction S.DS.DS was issuing a query for the subSchema and actually getting the results back, however it never got a proper LDAP response to the query. I would see the searchRequest and all the responses at the TCP layer (reassembled):

But I would never get a completed searchResEntry or searchResDone. Looking through the actual package bytes the very last one always stopped on the same schema attribute, so I asked the guys that managed this particular directory service about it and they had recently had a problem with that schema attribute and for some reason we were not getting a complete result at the network level from the directory server due to this schema corruption. The reason that S.DS.DS was able to render some attirbutes is that it knows that some schema objects are standardized and it knows what format they should be in. For example, it knows that cn, givenname, initials, and l are all strings which explains why they were rendered properly while everything else looked like a byte array. S.DS.DS didn't know how to render all of these other attributes because it has no cache file as a reference! The directory service admins restored the schema and S.DS.DS could now properly render the attributes to their proper type AND we all learned a little more about how System.DirectoryServices.DirectorySearcher works.

Paging Group Memberships from Active Directory using ADSI

First I’d like to get out of the way that this is my first real blog post.  I am not a regular blogger (clearly) and I don’t know how often I’ll truly be able to do this, but I desperately want to find ways to share my work and things that I run into on a daily basis so that perhaps others won’t have to struggle like I may have.

So that being said I think I’ve finally found something that I don’t see all that well documented that might be worth talking about.  Most people would use the Active Directory cmdlets to get group memberships (specifically Get-ADGroupMember or Get-ADGroup -Properties member -Identity groupName), but there are a couple of scenarios where this might not be ideal so I wanted to try to document how I page out group memberships using System.DirectoryServices objects and methods.  I’ve found a number of scenarios where Microsoft’s Active Directory cmdlets don’t fit my personal needs and so I’ve had to write my own AD cmdlets for a number of situations and this is one of them.

The first thing you’ll need to do is create a System.DirectoryServices.DirectorySearcher object.   Through MSDN’s documentation you’ll see there are a number of different constructors for a DirectorySearcher object (different ways you can create such an object with certain parameters).  To keep things simple for this post I’m going with an empty constructor.

$directorySearcher = New-Object System.DirectoryServices.DirectorySearcher

With the searcher created you need to fill out some of the properties before executing out search.  First we’ll point it at a specific domain controller, define a proper LDAP filter to find the group, and tell Active Directory what properties we want back:

$directorySearcher.SearchRoot = [ADSI]"LDAP://server01.domain.local"
$directorySearcher.Filer = '(&(objectClass=group)(cn=aBigGroup))'
[void]$directorySearcher.PropertiesToLoad.Add('cn')
[void]$directorySearcher.PropertiesToLoad.Add('distinguishedname')
[void]$directorySearcher.PropertiesToLoad.Add('member')

Truth be told there is a quicker way to do this with [ADSISearcher] type accelerator, however, at least in scripts, I prefer to spell things out in a way that is easy to read.  That being said now that we have our searcher object we can execute our search and store the results in a variable.

$results = $directorySearcher.FindAll()

Now and interesting thing has happened which you may (or may not) be aware of.  Normally when you make a request to Active Directory for certain properties, if they have values it will return those properties\attributes with the values included:

PS c:\> $results.properties.cn -as [String]
aBigGroup

The interesting thing about this is that if you look at the member property of this group it is empty!

PS c:\> $results.properties.member.count
0

Now I happen to know that this group has thousands of members.  The first clue here is to look to see what Active Directory actually returned to us.  If you look up above we specifically asked for cn, distinguishedname, and member.  However if you look at what comes back we got a little surprise:

PS c:\> $results.properties.PropertyNames
distinguishedname
member;range=0-1499
member
adspath
cn

So we did get back the properties we asked for but we got 2 additional ones.  The adspath is always going to come back with any object you get from Active Directory and is essentially the full path to the object (so this is expected).  The member;range=0-1499 however might be a little unexpected.  This is the clue to an aware ADSI client that it’s time to page out the membership.  If you look at the member;range=0-1499 property you’ll see that it has exactly 1,500 distinguished names in it:

PS c:\> $results.properties.'member;range=0-1499'.count
1500

What the above tells us is that:

1. This is a large group since the range property has been returned.
2. Active Directory is returning a maximum of 1,500 members at a time.

To get at the other members you will need to ask Active Directory for the next batch (1,500 at a time…thus the term “paging”).  To see the next batch you need to construct a few things (we’ll do it the long way first).

$members = New-Object System.Collections.ArrayList
$results.properties.'member;range=0-1499'.Foreach({ [void]$members.Add($psitem) })
$increment = $results.properties.'member;range=0-1499'.count -as [int]

$start = $increment
$end = $start + $increment -1
$memberProperty = 'member;range={0}-{1}' -f $start,$end
$directoryEntry = $results.Properties.adspath -as [String]

$memberPager = New-Object -TypeName System.DirectoryServices.DirectorySearcher -ArgumentList $directoryEntry,'(objectClass=*)',$memberProperty,'Base'
$pageResults = $memberPager.FindOne()

There are a couple things going on here worth explaining.  First I created an array list to store the members.  I use an array list (as opposed to an array) because they are a lot faster to add things to so this is more of a efficiency thing.  Next I load into the array the first batch of members that we already got back from AD and then I define an increment based on that initial count of members we got.  By default you will always see 1,500 members returned from AD unless you have custom LDAP policies, but I store that here as a number just to support different page sizes.  Next I define the start and end range.  If our first batch started with 0 and ended with 1499 then the next batch should start with 1500 and end with 2999 (i.e. 1500 + 1500 -1).  That gives us what is stored in $memberProperty= member:range=1500-2999.  Finally I pull out the full ADS path of the group and store it as a string since we’ll need it as a parameter for our next part.

Next we create another directory searcher object, but this time we structure it a little bit differently.  If you refer back to the MSDN documentation for System.DirectoryServices.DirectorySearcher you’ll see that there is a 4 parameter constructor and that’s what we’re using here.  We’re providing it the following:

1. A directory entry that is based off the adspath property that came back from our initial query which we're re-using for the paging.
2. An LDAP filter. In our case since our directory entry is to a specific object we keep the filter simple.
3. The member property we want to Active Directory to return to us.
4. Finally how far in the directory we should search. In this case since our directory entry is to a specific object we don’t need to search any further so we define it as just base.

With our pager object created we then need to initiate our query for members 1500-2999 by calling the FindOne() method.  I haven’t explained it yet, but at this point we know there are more so we just ask for one more.  Now we need to see what properties AD sent back to us, just like we had done earlier.

PS C:\> $pageResults.Properties.PropertyNames
member;range=1500-2999
adspath

In this case it gave back exactly what we were expecting so we need to add those members to our member array list:

$pageResults.Properties.'member;range=1500-2999'.Foreach({ [void]$members.Add($psitem) })

Now when I look at the number of members in the $members array list there are 3000.  But we’re not done as there are more members to pull out of Active Directory for this group (more on how we know shortly).  Like before we need to augment our start and end range by 1,500 members.  We stopped at 2999 so we need to start with 3000 and we need to end on 4,499 (3000 + 1500 - 1):

$start = $end + 1
$end = $start + $increment - 1
$memberProperty = 'member;range={0}-{1}' -f $start,$end

We create another DirectorySearcher object like we did before and pull down the next batch of memberships and add them to our array list.  Now you might be asking “how do we know when to stop?”.  Interestingly enough Active Directory will tell you.  In the example we’re working with the group has a little over 5,000 memberships.  When we ask Active Directory for members ranging from 4500-5999 when the results come back from our query the properties we get will by different than before:

PS C:\> $pageResults.Properties.PropertyNames
member;range=4500-*
adspath

So we asked for member;range=4500-5999 but we got back member;range=4500-*.  This is Active Directories way of telling us that this is the last batch. So now we do our last addition to the members array list:

$pageResults.Properties.'member;range=4500-*'.Foreach({ [void]$members.Add($psitem) })

So as you can see cmdlets such as Get-ADGroupMember or Get-ADGroup are doing quite a bit of heavy lifting for you, however if you are in a situation where you either can’t or don’t want to use the AD cmdlets to get memberships paging out the members isn’t really that hard once you understand the mechanics of how paging works.  Now what we did above is the …ahem… manual way and we are talking PowerShell here. We need to put this thing in a loop to do it right.

$directorySearcher = New-Object System.DirectoryServices.DirectorySearcher
$directorySearcher.SearchRoot = [ADSI]'LDAP://DC01.domain.local'
$directorySearcher.Filter = '(&(objectClass=group)(cn=ABigGroup))'
[void]$directorySearcher.PropertiesToLoad.Add('cn')
[void]$directorySearcher.PropertiesToLoad.Add('distinguishedname')
[void]$directorySearcher.PropertiesToLoad.Add('member')
$results = $directorySearcher.FindOne()

$members = New-Object System.Collections.ArrayList
if ($pageProperty = $results.Properties.PropertyNames.Where({$psitem -match '^member:range'}) -as [String]) {
    $directoryEntry = $results.Properties.adspath -as [String]
    $increment = $results.Properties.$pageProperty.count -as [Int]
    $results.Properties.$pageProperty.Foreach({ [void]$members.Add($psitem) })
    $start = $increment
    do {
        $end = $start + $increment - 1
        $memberProperty = 'member;range={0}-{1}' -f $start,$end
        $memberPager = New-Object -TypeName System.DirectoryServices.DirectorySearcher -ArgumentList $directoryEntry,'(objectClass=*)',$memberProperty,'Base'
        $pageResults = $memberPager.FindOne()
        $pageProperty = $pageResults.Properties.PropertyNames.Where({$psitem -match '^member:range'}) -as [String]
        $pageResults.Properties.$pageProperty.Foreach({ [void]$members.Add($psitem) })
    } until ( $pageProperty -match '^member.*\*$' )
}
else {
    $results.Properties.member.Foreach({ [void]$members.Add($psitem) })
}

Now all you’d need to do is wrap this up in a function\script, add in some parameters, and a dash of error checking with smart uses of try-catch and you have a group membership tool that doesn’t rely on any modules. Yay for one less dependency!  There is an alternative approach using System.DirectoryServices.AccountManagement, however it has a few problems for my normal uses (it's slower), but it does make it easier.  I may cover this in future posts.