Slow Directory Search

I have this code, if I pass in *text* it's very slow (2+ Mins),
however if I pass in text* it lightning fast (< 1 Second). Is there
something wrong with this code? The Directory has 50,000+ Person
Objects, way more Groups.

Is the key to just not allow searches that start with a wildcard?

Function GetMatches(ByVal FilterAttribute As String) As String

        Dim root As New
System.DirectoryServices.DirectoryEntry(m_sPath)
        Dim searcher As New
System.DirectoryServices.DirectorySearcher(root)
        Dim matches As System.Text.StringBuilder = New
System.Text.StringBuilder()
        Dim shtCount As Short

        searcher.Filter = "samaccountname=" & FilterAttribute
        searcher.PropertiesToLoad.Add("samaccountname")

        Dim results As SearchResultCollection
        results = searcher.FindAll
        Dim result As SearchResult

        shtCount = 0

        Try
            For Each result In results
 
matches.Append(result.GetDirectoryEntry.Properties("samaccountname").Value.ToString.ToUpper)
                matches.Append("|")
                shtCount = shtCount + 1
            Next
        Catch ex As System.Exception
            Return Nothing

        Finally
            searcher.Dispose()
            root.Dispose()
        End Try

        Return matches.ToString & CStr(shtCount)
    End Function