Skip Ribbon Commands
Skip to main content

Xadean's Empirical Musing

:

Quick Launch

Xadean's contribution to the consulting community through sharing his anthology of lesson's learned and technical insights.
January 29
Remote Desktop Connection Authentication Failure with Windows Hello
On a Wndows 11 Enterprise PC attempting to connect to a virtual machine (VM) that has a Windows 10 Enterprise operating system installed where both systems are Azure AD domain joined or registered, encountering error.

[Window Title]
Remote Desktop Connection 

[Content]
An authentication error has occurred (Code: 0x8007013d).

Remote computer: 192.168.1.206

An authentication error has occurred (Code: 0x8007013d). Error code: 0x0

[^] Hide details  [OK]

[Expanded Information]
Error code: 0x0
Extended error code: 0x0
Timestamp (UTC): 01/28/23 03:26:30 PM
Activity ID: e1b633a1-336f-4578-a453-4c5d17550000

Press Ctrl+C to copy.

The RDP client was prompting for facial recognition or entering a PIN to authenticate with Windows 11 enabled.

RESOLUTION:
Authenticate with a local administrator account on the remote VM.  In my case, I enabled the local Administrator account and set the password.  Then I used the local Administrator account to authenticate​.  After that was successful, the VM prompted me to logon to Windows with the PIN and it worked.
June 17
Skype for Business Edge Server Not Able to Communicate via Federation to Microsoft Office 365 (Teams and Cloud Voice Mail Auto Attendants Not Reachable)

Issue Description and Background:

We have an enterprise voice deployment of Skype for Business Server 2015 integrated with Microsoft (MS) Teams Phone System running in a hybrid configuration with direct routing using a Ribbon Communications SBC. Inbound calls to our main phone number are answered by a MS Teams auto attendant (AA). However, that suddenly stopped working after we had incidentally recently deployed Direct Routing for MS Teams and afterwards we noticed the AA's were no longer reachable via on-premises Skype for Business Server 2015 via federation.

   

The following errors were in the Event Viewer logs on the Edge Server under Lync Server.

   

EVENT ID: 14428

TLS outgoing connection failures.

   

Over the past 66 minutes, Skype for Business Server has experienced TLS outgoing connection failures 31 time(s). The error code of the last failure is 0x800B0109(CERT_E_UNTRUSTEDROOT) while trying to connect to the server "sipfed.online.lync.com" at address [52.112.65.203:5061], and the display name in the peer certificate is "sipfed.online.lync.com".

Cause: Most often a problem with the peer certificate or perhaps the host name (DNS) record used to reach the peer server. Target principal name is incorrect means that the peer certificate does not contain the name that the local server used to connect. Certificate root not trusted error means that the peer certificate was issued by a remote CA that is not trusted by the local machine.

Resolution:

Check that the address and port matches the FQDN used to connect, and that the peer certificate contains this FQDN somewhere in its subject or SAN fields. If the FQDN refers to a DNS load balanced pool then check that all addresses returned by DNS refer to a server in the same pool. For untrusted root errors, ensure that the remote CA certificate chain is installed locally. If you have already installed the remote CA certificate chain, then try rebooting the local machine.

   

   

Multiple invalid incoming certificates.

   

In the past 147 minutes the server received 1 invalid incoming certificates. The last one was from host 52.112.67.51.

Cause: This can happen if a remote server presents an invalid certificate due to an incorrect configuration or an attacker.

Resolution:

No action needed unless the number of failures is large. Contact the administrator of the host sending the invalid certificate and resolve this problem.

   

   

We also captured a SIP trace of a test inbound call that was failing using the ClsLogging tool and noticed the following error.

   

TL_INFO(TF_PROTOCOL) [LYNCPOOL\btfls13fe]1F64.347C::06/13/2022-23:20:45.600.000133D2 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(261)) [2430505428]

Trace-Correlation-Id: 2430505428

Instance-Id: 18C69

Direction: incoming

Peer: LSEDGEPOOL.hq.biztechfusion.com:5061

Message-Type: response

Start-Line: SIP/2.0 504 Server time-out

From: "AHMASI XADEAN"<sip:12024891495;phone-context=PstnGateway_10.1.20.3@biztechfusion.com;user=phone>;epid=8D00A9C65B;tag=937ee3bfb9

To: <sip:4433600959;phone-context=PstnGateway_10.1.20.3@biztechfusion.com;user=phone>;tag=11BF95CFA045B9B046D298304CD0978C

Call-ID: 292742da-6c2f-47f0-b881-f4f6379787e9

CSeq: 5797 INVITE

Via: SIP/2.0/TLS 10.1.20.33:49948;branch=z9hG4bKAEF14639.AB56DBC7A7D924DD;branched=FALSE;ms-received-port=49948;ms-received-cid=1E4C00

Via: SIP/2.0/TLS 10.1.20.33:53216;branch=z9hG4bKd3415c7;ms-received-port=53216;ms-received-cid=26A800

Content-Length: 0

ms-diagnostics: 1010;reason="Certificate trust with another server could not be established";ErrorType="The peer certificate is not chained off a trusted root";tls-target="sipfed.online.lync.com";PeerServer="sipfed.online.lync.com";HRESULT="0x800B0109(CERT_E_UNTRUSTEDROOT)";source="LYNCSIP.biztechfusion.com"

ms-edge-proxy-message-trust: ms-source-type=EdgeProxyGenerated;ms-ep-fqdn=LSEDGEPOOL.hq.biztechfusion.com;ms-source-verified-user=verified

Server: RTC/6.0

$$end_record

   

   

   

The test call to 4433600959 is translated to +7006 (Teams Auto Attendant), then the Skype FE/Med Srvr sends the call to the Edge Srvr to get to O365 Cloud Voice Mail / Teams Auto Attendant, and gets a Cert error. Looks like it doesn't trust the O365 side server's cert, for some reason.

   

RESOLUTION:

The error messages was clearly indicating that our Edge Server was not trusting the SSL certificate being presented by Microsoft's Edge Servers (sipfed.online.lync.com). Therefore, we had to do the following to update the trusted root CA store on the Edge Server.

   

Open an elevated command prompt and enter the following commands:

   

# The following command downloads the Trusted Root CA's to a serialized store file:

Certutil -generateSSTFromWU WURoots.sst

   

# The following command opens the serialized store file (you can also simply double-click on the file to open it):

start explorer.exe wuroots.sst

   

Once the serialized store file is open, expand to the lowest level "Certificates" folder in the left navigation pane, and then you will see certificates for all of the trusted root certification authorities in the right action pane.

   

   

Next, you will need to open the Microsoft Management Console (right-click on start menu and select run, then type mmc and click OK) and add the Certificates snap-in for your Local Machine. Navigate to the Trusted Root Certification Authorities and right-click the Certificates subfolder in the left navigation pane.

   

   

Select "All Tasks" from the context menu and then "Import". Click next to progress through the wizard to browse and select the Microsoft Certificate Serialized Store (.SST) file you created and finish to complete importing all the root CA certs.

   

After importing the certs, federation between our Edge and Microsoft (sipfed.online.lync.com) began working immediately.

   

You may also determine whether or not you would like to update your registry on the server to automatically update the trusted root certification authorities by performing the following action.

   

   

Save the following to a text file and save it as "AuthRoot.reg":

   

BEGIN FILE [Do Not Include in the Text File]

   

Windows Registry Editor Version 5.00

   

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot]

"DisableRootAutoUpdate"=dword:00000000

"EnableDisallowedCertAutoUpdate"=dword:00000001

   

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot\Certificates]

   

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot\CRLs]

   

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot\CTLs]

   

END FILE [Do Not Include in the Text File]

   

Now, double-click on the AuthRoot.reg file to configure the registry accordingly.

   

March 23
Issue with Skype for Business Server Conferencing Not Working

Overview of Environment

  • Hyper-V virtualized environment hosted on Storage Spaces Direct Windows Server 2019 cluster
  • Skype for Business (SfB) Server 2015 Enterprise Pool (Single Server Member) with Front End and Mediation Roles Collocated
  • SfB Edge Pool with Single Server Member
  • Ribbon Communications (formerly Sonus) SBC
  • Centralized SIP Trunking for PSTN integration
  • All workloads deployed including Enterprise Voice
  • Hybrid configuration with Microsoft Teams operating in Islands mode

Description of Issue Encountered

All SfB Server users both internal and remote reported not being able to join any audio/video conferences. Users could start a Meet Now conference, but no other users could join. No users could dial-in to a conference using the Meeting ID via PSTN. Users could call out to an external phone number and connect successfully. However, they could not add another participant to that call (3-way).

The following are the Lync Server event log errors.

User failed to join the conference.

Microsoft.Rtc.Collaboration.ConferenceFailureException:The operation failed due to a response from the server. For more information, examine the properties on the exception and inner exception.

at Microsoft.Rtc.Signaling.SipAsyncResult`1.ThrowIfFailed()

at Microsoft.Rtc.Signaling.Helper.EndAsyncOperation[T](Object owner, IAsyncResult result)

at Microsoft.LiveServer.Caa.CaaCall.EndTransfer(IAsyncResult asyncResult, Boolean& retry, Exception& caught)

Detected at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

at System.Environment.get_StackTrace()

at Microsoft.Rtc.Collaboration.ConferenceFailureException..ctor(String message, Exception innerException)

at Microsoft.Rtc.Collaboration.Conferencing.SendCommandAsyncResult.CreateConferenceFailureException(ConferenceCommandResponse commandResponse, Exception innerException)

at Microsoft.Rtc.Collaboration.Conferencing.SendCommandAsyncResult.ProcessCccpResponse(SipMessageData messageData, responsetype response, Boolean& isPendingResponse)

at Microsoft.Rtc.Collaboration.Conferencing.SendCommandAsyncResult.ProcessStatusMessage(SipMessageData statusMessageData, responsetype response)

at Microsoft.Rtc.Collaboration.Conferencing.StatusMessageReceivedWorkItem.Process()

at Microsoft.Rtc.Signaling.AsyncWorkitemQueue.ProcessItems()

at Microsoft.Rtc.Signaling.QueueWorkItemState.ExecuteWrappedMethod(WaitCallback method, Object state)

at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

at System.Threading.ThreadPoolWorkQueue.Dispatch()

Cause: Administration issues.

Resolution:

Verify that Conferencing Attendant is installed in a supported topology and the dependant Front End servers are functioning correctly.

 

The Audio/Video Conferencing Server cannot add Conferencing Announcement Service to a conference.

Conferencing Announcement Service failing: sip:LYNCPOOL.biztechfusion.com@biztechfusion.com;gruu;opaque=srvr:Microsoft.Rtc.Applications.Cas:k1Mdhi4-FVSxbeCSuouZogAA Diagnostic Message: Not Available

Cause: The Audio/Video Conferencing Server is not able to reach the Conferencing Announcement Service. This may be due to network connectivity issues or unavailability of the Conferencing Announcement Service. Conference Announcements in audio/audio-video conferences will be unavailable due to this issue. PSTN dial in users will be impacted, as they will be unable to exercise DTMF controls and other PSTN dial in conferencing functionality while in the conference.

Resolution:

Resolution=Please ensure network connectivity and availability of the Conferencing Announcement Service for the Audio/Video Conferencing Server to be able to function correctly. For additional information to troubleshoot the issue, please look at the events AVMCU_E_CAS_UNREACHABLE (Event ID: 32083).

 

The Audio/Video Conferencing Server cannot add Conferencing Announcement Service to a conference.

Audio/Video Conferencing Server Pool: btfls13fe.hq.biztechfusion.com; Conferencing Announcement Service: sip:LYNCPOOL.biztechfusion.com@biztechfusion.com;gruu;opaque=srvr:Microsoft.Rtc.Applications.Cas:k1Mdhi4-FVSxbeCSuouZogAA; Diagnostic Message: Error Message: The server encountered an unexpected internal error Sip Response Code: 500

Cause: The Audio/Video Conferencing Server is not able to reach the Conferencing Announcement Service. This may be due to network connectivity issues or unavailability of the Conferencing Announcement Service. Conference Announcements in audio/audio-video conferences will be unavailable due to this issue. PSTN dial in users will be impacted, as they will be unable to exercise DTMF controls and other PSTN dial in conferencing functionality while in the conference.

Resolution:

Please ensure network connectivity and availability of the Conferencing Announcement Service for the Audio/Video Conferencing Server to be able to function correctly.

Resolution:

After disabling conferencing in Topology Builder in the properties of the Front End Pool, publishing the topology, and running the bootstrapper.exe program to remove the conferencing features, then reinstalling the conferencing features using the same preceding steps, the issue remained unresolved.

Next, downloaded the latest Cumulative Update (CU) and hotfixes for SfB. Installed the CU and still the issue remained unresolved.

Checked the assignment of SSL certificates to the collocated Front End Server and noticed that the same SSL certificate issued by a public Certificate Authority (CA) with Subject Alternative Names has been assigned to the Default Certificate hierarchy that includes Server default, We services internal, and Web services external. Changing the SSL certificate assignment for Server default using SfB Deployment Wizard to one that contained only the pool name, server name, and sip.domain.name as SANs. After restarting the SfB services, conferencing started working again.

November 04
Can't sign in to Skype for Business after ADAL (Modern Authentication) is Enabled

Environment Details:

  • Hybrid deployment of Skype for Business (SfB) Server 2015 Enterprise Pool with all workloads enabled including Enterprise Voice with on-premises PSTN integration.
  • All user mailboxes homed on Exchange Online (ExO).
  • ADAL / Modern Authentication enabled for SfB and ExO (specifically HMA).

Description:

After executing "Set-CsOAuthConfiguration -ClientAuthorizationOAuthServerIdentity evoSTS" to enable HMA on SfB, SfB desktop clients began ecountering the following error message.

Error Message: Can't sign in to Skype for Business. You didn't get signed in. It might be your sign-in address or logon credentials, so try those again.

Resolution:

Followed Method 1 of this reference article: https://support.microsoft.com/en-us/topic/cannot-sign-in-to-skype-for-business-after-enable-adal-aka-modern-authentication-4f6f9618-0abf-1e3d-0ada-e6b36d137170.

To resolve this issue, change the Internet Explorer "User Authentication" settings on the affected client computers to "prompt for user name and password" in the security zone. To do this, use one of the following methods.

Method 1: Change the setting manually

  1. In Internet Explorer, click Tools, click Internet Options, and then click the Security tab.
  2. Select the security zone that includes the STS URL. Typically, this is the Local Intranet zone.
  3. Click the Custom level button, and then scroll to the end of the Settings list.
  4. In the User Authentication section, select the Prompt for user name and password option.
August 03
Modify Exchange Migration Throttling Policies

Creating a New, Non-Default Throttling Policy

To create a new, non-default throttling policy, run the New-ThrottlingPolicy cmdlet and set the parameters you want. Any parameters that you omit will inherit the values from the default throttling policy. The following example creates a new throttling policy, ClientThrottlingPolicy2. The new policy has nearly the same settings as the default throttling policy. The difference is that the new non-default throttling policy, ClientThrottlingPolicy2, sets EWSPercentTimeInCAS to 80 and turns off EWSPercentTimeInAD throttling.

New-ThrottlingPolicy -Name ClientThrottlingPolicy2 -EWSPercentTimeInCAS 80 -EWSPercentTimeInAD $null;

Get-ThrottlingPolicy | where-object {$_.IsDefault -eq $true}

 

Connect to Exchange Online using below commands:

$UserCredential = Get-Credential

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection

Import-PSSession $Session

 

Once connected run below logs to collect the output for migration statuses:

 

Start-Transcript C:\Temp\MigrationSlow.txt

Get-MigrationBatch –Identity <name of the migration batch> | fl *status*

Get-MigrationUser -Identity <user_mailbox_alias>

Get-MigrationUser -Identity <user_mailbox_alias>

Get-MoveRequestStatistics <job_1_name> -IncludeReport -DiagnosticInfo verbose | Export-CliXml C:\Temp\MoveReportUser1.xml

Get-MoveRequestStatistics <job_2_name> -IncludeReport -DiagnosticInfo verbose | Export-CliXml C:\Temp\MoveReportUser2.xml

Stop-Trancript

 

For Exchange 2010 MRS Proxy Enabled CAS-role Servers

If the production Exchange 2010 CAS are being used as MRS Proxies:

All Exchange 2010 CAS that are in the load balanced namespace are required to be enabled for MRS Proxy. For example, if the MRS Proxy endpoint is published to the internet as https://webmail.contoso.com, and that namespace is loadbalanced, then all Exchange 2010 CAS that participate in that load balanced configuration would have to have the MRS Proxy capability enabled on them,

Several settings can reduce performance for mailbox moves. Editing the MSExchangeMailboxReplication.exe.config file, located on all CAS servers in the same folder where Exchange is installed: \Program Files\Microsoft\Exchange Server\V14\Bin\MSExchangeMailboxReplication.exe.config. and modifying the following parameters to higher values can help improve migration throughput if an Exchange 2010 CAS is enabled as an MRS Proxy:

  1. MaxActiveMovesPerSourceMDB - This property specifies the total number of tasks MRS can perform that involve the mailbox database as a data source. Types of tasks include moving mailboxes located on the database, exporting mailbox data from mailboxes located on the database, and restoring mailbox data from the database. You can specify a value from 0 through 100. The default value is 5 concurrent tasks. Consider raising to 10
  2. MaxActiveMovesPerTargetMDB - This property specifies the total number of tasks MRS can perform that involve the mailbox database as a data target. Types of tasks include moving mailboxes to the database, importing mailbox data into a mailbox located on the database, and restoring mailbox data to a mailbox located on the database. You can specify a value from 0 through 100. The default value is 2 concurrent tasks. Consider raising to 10 or higher.
  3. MaxActiveMovesPerSourceServer - This property specifies the total number of tasks MRS can perform that include the server as a data source. You can specify a value from 0 through 1000. The default value is 50 concurrent moves. I don't see this as an issue, but mentioning it for thoroughness.
  4. MaxActiveMovesPerTargetServer - This property specifies the total number of tasks MRS can perform that involve the server as a data target. You can specify a value from 0 through 1000. The default value is 5 concurrent moves. Consider raising to 10

 

References

Understanding Client Throttling Policies: https://docs.microsoft.com/en-us/previous-versions/office/exchange-server-2010/dd297964(v=exchg.141)

https://blogs.technet.microsoft.com/vytas/2018/03/02/maximizing-and-optimizing-your-mailbox-move-data-throughput-for-on-premise-to-o365-and-exchange-online-migrations/

 

August 03
How to Make Exchange Legacy Public Folders Visible to Exchange Online Users in a Hybrid Configuration

Issue Description

In an environment that had a pre-existing Exchange Server 2010 SP3 (multiple distributed servers running Mailbox (M), Hub Transport (HT), Client Access (CAS), and Unified Messaging (UM) roles) and new configured hybrid integration with Exchange Online (ExO) in Office 365 tenant, migrated mailbox users were not able to access / view any of the legacy public folders homed on-premises in Exchange 2010.

Resolution

Here are the troubleshooting steps that we have done to make the legacy Exchange 2010 public folders visible to Office 365 ExO migrated mailbox users.

Step 1: Make remote public folders discoverable

 

If your public folders are on Exchange 2010 or later servers, you must install the Client Access server (CAS) role on all mailbox servers that have a public folder database. This allows the Microsoft Exchange RpcClientAccess service to be running so that all clients can access public folders. For more information, see Install Exchange Server 2010.

 

Note: This server doesn't have to be part of the Client Access load balancing. For more information, see Understanding Load Balancing in Exchange 2010.

 

  • Create an empty mailbox database on each public folder server. For Exchange 2010, run the following command. This command excludes the mailbox database from the mailbox provisioning load balancer. This prevents new mailboxes from being added automatically to this database.

 

New-MailboxDatabase -Server <PFServerName_with_CASRole> -Name <NewMDBforPFs> -IsExcludedFromProvisioning $true

 

Note: We recommend that the only mailbox that you add to this database is the proxy mailbox that you'll create in step 3. No other mailboxes should be created on this mailbox database.

 

  • Create a proxy mailbox within the new mailbox database and hide the mailbox from the address book. The SMTP of this mailbox will be returned by AutoDiscover as the DefaultPublicFolderMailbox SMTP, so that by resolving this SMTP the client can reach the legacy exchange server for public folder access.

 

New-Mailbox -Name <PFMailbox1> -Database <NewMDBforPFs>

Set-Mailbox -Identity <PFMailbox1> -HiddenFromAddressListsEnabled $true

   

  • For Exchange 2010, enable AutoDiscover to return the proxy public folder mailboxes.

 

Set-MailboxDatabase <NewMDBforPFs> -RPCClientAccessServer <PFServerName_with_CASRole>

 

  • Repeat the preceding steps for every public folder server in your organization.

 

Example: 

New-MailboxDatabase -Server SRVR01 -Name PFMailboxDB -IsExcludedFromProvisioning $true

New-MailboxDatabase -Server SRVR02 -Name PFMailboxDB2 -IsExcludedFromProvisioning $true        

   

New-Mailbox -Name PFMailbox1 -Database PFMailboxDB

Set-Mailbox -Identity PFMailbox1 -HiddenFromAddressListsEnabled $true

   

New-Mailbox -Name PFMailbox2 -Database PFMailboxDB2

Set-Mailbox -Identity PFMailbox1 -HiddenFromAddressListsEnabled $true

   

Set-MailboxDatabase PFMailboxDB -RPCClientAccessServer SRVR01

Set-MailboxDatabase PFMailboxDB2 -RPCClientAccessServer SRVR02

   

  • Run the below command on exchange online PowerShell to make the public folder as remote and add the name of on-premises public folder in remote.

    Set-OrganizationConfig -PublicFoldersEnabled Remote -RemotePublicFolderMailboxes PFMailbox1,PFMailbox2,PFMailbox******NAMES OF ALL PUBLIC FOLDER****"

     

  • Run the full sync on sync server:

    Start-ADSyncSyncCycle -PolicyType Initial

    Note: This process take up to 3 hours to replicate.

     

  • For Exchange 2010, enable AutoDiscover to return the proxy public folder mailboxes.

 

Set-MailboxDatabase <NewMDBforPFs> -RPCClientAccessServer <PFServerName_with_CASRole>

 

  • Run the following command in ExO Powershell.

     

    Get-casmailbox –resultsize unlimited | set-casmailbox – PublicFolderClientAccess $true

     

    Note: This is a critically important step. You may also configure individual users to access the legacy on-premises public folder by executing the following ExO Powershell command.

     

    Set-CASMailbox <user_mailbox_alias> -PublicFolderClientAccess $true

    Get-CASMailbox <user_mailbox_alias> | fl *public*

 

References

https://docs.microsoft.com/en-us/exchange/collaboration-exo/public-folders/set-up-legacy-hybrid-public-folders

 

Migrate Ex2010 Legacy Public Folders to O365

https://docs.microsoft.com/en-us/exchange/collaboration-exo/public-folders/batch-migration-of-legacy-public-folders

 

Migrate Ex2010 Public Folders to O365 Groups

https://docs.microsoft.com/en-us/exchange/use-batch-migration-to-migrate-exchange-2010-public-folders-to-office-365-groups-exchange-2013-help

   

https://technet.microsoft.com/en-us/library/dn912663%28v=exchg.150%29.aspx?f=255&MSPPError=-2147217396#Scripts 

   

https://thoughtsofanidlemind.com/2013/12/13/migration-modern-public-folders/

   

https://technet.microsoft.com/en-us/library/dn986544(v=exchg.150).aspx

https://blogs.technet.microsoft.com/exchange/2015/11/02/running-powershell-cmdlets-for-large-numbers-of-users-in-office-365/

June 18
Cannot Activate Windows Server

Issue: Not able to Activate Windows Server 2019 (Datacenter & Standard)

 

Resolution: Execute the following commands from an elevated command prompt:

 

(For Windows Server 2019)

    slmgr /ipk "N6FCQ-4DRBT-F2W2G-TPMVG-V6F77"

    slmgr /ato

    

(For older versions of Windows Server)

    slui 3

February 27
CMS Replication (XDS) Not Working Between Front End & Edge Servers After Disabling Weak Protocols & Ciphers

Microsoft Case #: 13231981

Created on: Thursday, February 21, 2019

Support request number: 119022124001575

Product: Skype for Business Server 2015 (on-premises)

Issue: CMS Replication (XDS) Not Working Between Front End & Edge Servers After Disabling Weak Protocols & Ciphers

Description: Output of "Get-CsManagementReplicationStatus" shows false in UpToDate field for all Edge Servers in the on-premises deployment following execution of "Invoke-CsManagementStoreReplication" PS cmdlets. LastUpdateCreation shows a date in July 2018 of last year. Skype for Business (SFB) Control Panel showing red X next to Edge Servers in Topology. After capturing a debug trace of the replication process with CLSLogger tool, observed the following error message:

TL_WARN(TF_COMPONENT) [FEPoolNAME\FEServerNAME]1E14.0AA8::02/15/2019-19:29:20.931.00002004 (XDS_File_Transfer_Agent,FileTransferTask.CopyFilesFromReplicaUsingWcf:filetransfertask.cs(755)) (0000000002FFC0E0)[FileTransferTask(6, 2/15/2019 11:26:49 AM): {TASK_NOT_STARTED, fromReplica, [FEServerNAME.fqdn, HttpsWebService, 4443], 0}] Failed to copy files from replica. Exception: [System.ServiceModel.CommunicationException: An error occurred while making the HTTP request to https:// FEServerNAME.fqdn:4443/ReplicationWebService. This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case. This could also be caused by a mismatch of the security binding between the client and the server. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Also noticed the following error messaging after executing "Get-CsPoolFabricState -PoolFQDN FEPoolName.FQDN" PS cmdlet:

PS C:\Users\xahmasi> Get-CsPoolFabricState -PoolFqdn FEPoolName.FQDN

Get-CsPoolFabricState : An error occurred while receiving the HTTP response to

https:// FEServerName.FQDN /LiveServer/UserPinManagement/FabricManagement/. This could be due to the service endpoint

binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server

(possibly due to the service shutting down). See server logs for more details.

At line:1 char:1

+ Get-CsPoolFabricState -PoolFqdn FEPoolName.FQDN

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : InvalidOperation: (:) [Get-CsPoolFabricState], CommunicationException

+ FullyQualifiedErrorId : Error getting fabric state. For details, see inner exception.,Microsoft.Rtc.Management.H

ADR.FabricState.GetOcsPoolFabricStateCmdlet

Root Cause: The Windows Communications Framework (WCF) does not by default use TLS 1.1 or 1.2.  Instead, it uses TLS 1.0 and RC4 ciphers (specifically TLS_RSA_WITH_RC4_128_SHA).  These weak protocols (i.e. SSL 2.0/3.0 and TLS 1.0) and ciphers (i.e. RC2, RC4, DES, and 3DES) had been disabled due to governance compliance.

Resolution: In order to fix, under following keys for all the version key listed like V1.0, V2.0.50727,v3,v4.0.30319, create key "SchUseStrongCrypto" (type DWORD, value 1):

HKLM\software\Wow6432Node\Microsoft\.NETFramework\

HKLM\software\microsoft\,NETFramework\

After adding this key to and rebooting all Edge / Front End Servers and executing invoke-CsManagementReplicationStatus, the UpToDate status on all Edge / Front End Servers is showing True and there is a green check next to all Edge  / Front End Servers in the Skype for Business Control Panel under Topology.

Reference(s): https://blogs.msdn.microsoft.com/benjaminperkins/2014/11/04/using-tls-1-2-with-wcf/

 

 

 

November 26
Skype for Business (SFB) Client Drops from A/V Web Conferences Immediately after Joining

Description of Issue:

In an environment with the following on-premises elements installed, a single user kept experiencing getting dropped from SFB Online Meetings immediately after joining with audio.

 

  • Windows 2012 R2 Active Directory
  • Enterprise SFB 2015 Front End Pool
  • Enterprise SFB 2015 Edge Pool
  • SFB 2015 Persistent Chat Server
  • Office Web Apps Server
  • Windows 2012 R2 Web Application Proxy
  • Windows 2016 IIS ARR

 

The user began to experience this issue following a perimeter firewall upgrade and laptop rebuild. Ergo, there were several potential culprits as a root cause.

 

Root Cause Analysis & Resolution:

Since the firewall was recently upgraded, the initial thinking was the inbound access policies could be misconfigured and causing the user to be dropped. The peculiar thing was that other remote users did not experience the same issue. The assumption was that all users should be experiencing the same behavior if the firewall was the issue. Nonetheless, we configured a packet sniffer on the firewall and captured traces from the user with the issue and other users without the issue to compare the traffic flow. All traces showed the clients connecting to the SIP Access, Web Conferencing, and A/V Conferencing Edge interfaces as expected. We did notice that the user with the error was sending TCP rst and ICMP requests to the A/V Conferencing Edge interface right before being disconnected. Ruled the firewall out since it was accepting all the inbound traffic.

 

Analyzed the PC of the user with the issue beginning with adding exceptions to Windows Defender for the Lync.exe application. Disabled the firewall and real-time protection on the PC. Still the user experienced the issue. Had the user join another organization's SFB online meeting and the user did not experience the disconnect behavior. Looked at the SFB Edge Pool configuration and noticed that there is a SSL certificate assigned that was issued by an internal Certificate Authority (CA) Server. Exported the internal CA Server's root certificate and imported it into the Trusted Root Certificate Authorities store on the user's PC. VOILA! That solved the issue (importing the internal CA's root certificate).

November 05
Exchange Server 2013 Management Tools (EMS & EAC) Failing with Errors

Issue Experienced:

When attempting to launch Exchange Management Shell (EMS) and Exchange Admin Center (EAC), running into the following error messages respectively.

EMS Error #1:

 

Welcome to the Exchange Management Shell!

 

Full list of cmdlets: Get-Command

Only Exchange cmdlets: Get-ExCommand

Cmdlets that match a specific string: Help *<string>*

Get general help: Help

Get help for a cmdlet: Help <cmdlet name> or <cmdlet name> -?

Exchange team blog: Get-ExBlog

Show full output for a command: <command> | Format-List

 

Show quick reference guide: QuickRef

Tip of the day #63:

 

Any cmdlet that accepts a size value lets you specify whether the integer value is in kilobytes (KB), megabytes (MB), gi

gabytes (GB), or terabytes (TB). For example:

 

Set-Mailbox "Kim Akers" -ProhibitSendQuota 200MB -UseDatabaseQuotaDefaults $False

 

VERBOSE: Connecting to ExchSrvr.mail.com.

New-PSSession : [ExchSrvr.mail.com] Connecting to remote server ExchSrvr.mail.com failed with the

following error message : <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<title>IIS 8.0 Detailed Error - 500.0 - Internal Server Error</title>

<style type="text/css">

<!--

body{margin:0;font-size:.7em;font-family:Verdana,Arial,Helvetica,sans-serif;}

code{margin:0;color:#006600;font-size:1.1em;font-weight:bold;}

.config_source code{font-size:.8em;color:#000000;}

pre{margin:0;font-size:1.4em;word-wrap:break-word;}

ul,ol{margin:10px 0 10px 5px;}

ul.first,ol.first{margin-top:5px;}

fieldset{padding:0 15px 10px 15px;word-break:break-all;}

.summary-container fieldset{padding-bottom:5px;margin-top:4px;}

legend.no-expand-all{padding:2px 15px 4px 10px;margin:0 0 0 -12px;}

legend{color:#333333;;margin:4px 0 8px -12px;_margin-top:0px;

font-weight:bold;font-size:1em;}

a:link,a:visited{color:#007EFF;font-weight:bold;}

a:hover{text-decoration:none;}

h1{font-size:2.4em;margin:0;color:#FFF;}

h2{font-size:1.7em;margin:0;color:#CC0000;}

h3{font-size:1.4em;margin:10px 0 0 0;color:#CC0000;}

h4{font-size:1.2em;margin:10px 0 5px 0;

}#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS",Verdana,sans-serif;

color:#FFF;background-color:#5C87B2;

}#content{margin:0 0 0 2%;position:relative;}

.summary-container,.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}

.content-container p{margin:0 0 10px 0;

}#details-left{width:35%;float:left;margin-right:2%;

}#details-right{width:63%;float:left;overflow:hidden;

}#server_version{width:96%;_height:1px;min-height:1px;margin:0 0 5px 0;padding:11px 2% 8px 2%;color:#FFFFFF;

background-color:#5A7FA5;border-bottom:1px solid #C1CFDD;border-top:1px solid #4A6C8E;font-weight:normal;

font-size:1em;color:#FFF;text-align:right;

}#server_version p{margin:5px 0;}

table{margin:4px 0 4px 0;width:100%;border:none;}

td,th{vertical-align:top;padding:3px 0;text-align:left;font-weight:normal;border:none;}

th{width:30%;text-align:right;padding-right:2%;font-weight:bold;}

thead th{background-color:#ebebeb;width:25%;

}#details-right th{width:20%;}

table tr.alt td,table tr.alt th{}

.highlight-code{color:#CC0000;font-weight:bold;font-style:italic;}

.clear{clear:both;}

.preferred{padding:0 5px 2px 5px;font-weight:normal;background:#006633;color:#FFF;font-size:.8em;}

-->

</style>

 

</head>

<body>

<div id="content">

<div class="content-container">

<h3>HTTP Error 500.0 - Internal Server Error</h3>

<h4>Module &quot;WSMan&quot; could not be found</h4>

</div>

<div class="content-container">

<fieldset><h4>Most likely causes:</h4>

<ul> <li>The module could not be found.</li> <li>IIS received the request; however, an internal error

occurred during the processing of the request. The root cause of this error depends on which module handles the

request and what was happening in the worker process when this error occurred.</li> <li>IIS was not able to access

the web.config file for the Web site or application. This can occur if the NTFS permissions are set incorrectly.</li>

<li>IIS was not able to process configuration for the Web site or application.</li> <li>The authenticated user

does not have permission to use this DLL.</li> </ul>

</fieldset>

</div>

<div class="content-container">

<fieldset><h4>Things you can try:</h4>

<ul> <li>Verify that the module name is not misspelled in the configuration/system.webServer/globalModules

config section. The error description may contain additional information to help you determine which module is causing

the error.</li> <li>Ensure that the NTFS permissions for the web.config file are correct and allow access to the

Web server's machine account.</li> <li>Check the event logs to see if any additional information was logged.</li>

<li>Verify the permissions for the DLL.</li> <li>Create a tracing rule to track failed requests for this HTTP

status code. For more information about creating a tracing rule for failed requests, click <a

href="http://go.microsoft.com/fwlink/?LinkID=66439">here</a>. </li> </ul>

</fieldset>

</div>

 

<div class="content-container">

<fieldset><h4>Detailed Error Information:</h4>

<div id="details-left">

<table border="0" cellpadding="0" cellspacing="0">

<tr class="alt"><th>Module</th><td>&nbsp;&nbsp;&nbsp;IIS Web Core</td></tr>

<tr><th>Notification</th><td>&nbsp;&nbsp;&nbsp;BeginRequest</td></tr>

<tr class="alt"><th>Handler</th><td>&nbsp;&nbsp;&nbsp;Not yet determined</td></tr>

<tr><th>Error Code</th><td>&nbsp;&nbsp;&nbsp;0x80070002</td></tr>

 

</table>

</div>

<div id="details-right">

<table border="0" cellpadding="0" cellspacing="0">

<tr class="alt"><th>Requested URL</th><td>&nbsp;&nbsp;&nbsp;https://ExchSrvr.mail.com:444/powershell?serial

izationLevel=Full;ExchClientVer=15.0.1395.4;clientApplication=ManagementShell;TargetServer=;PSVersion=3.0&amp;sessionID

=Version_15.0_(Build_1394.4)=rJqNiZqNgaqsuqe8t8/O0ZyQkYuWkYqKkpGai9GckJKBzsbLzsbGycbOyoHNz87H0s7O0s/Kq8/LxczJxc3H</td><

/tr>

<tr><th>Physical Path</th><td>&nbsp;&nbsp;&nbsp;C:\Program Files\Microsoft\Exchange

Server\V15\ClientAccess\PowerShell-Proxy</td></tr>

<tr class="alt"><th>Logon Method</th><td>&nbsp;&nbsp;&nbsp;Not yet determined</td></tr>

<tr><th>Logon User</th><td>&nbsp;&nbsp;&nbsp;Not yet determined</td></tr>

 

</table>

<div class="clear"></div>

</div>

</fieldset>

</div>

 

<div class="content-container">

<fieldset><h4>More Information:</h4>

This error means that there was a problem while processing the request. The request was received by the Web server,

but during processing a fatal error occurred, causing the 500 error.

<p><a href="http://go.microsoft.com/fwlink/?LinkID=62293&amp;IIS70Error=500,0,0x80070002,9200">View more information

&raquo;</a></p>

<p>Microsoft Knowledge Base Articles:</p>

<ul><li></li></ul>

 

</fieldset>

</div>

</div>

</body>

</html>

Error occurred during the Kerberos reponse.

[Server=ExchSrvr, TimeStamp = 114/2018 23:28:18]

For more information, see the about_Remote_Troubleshooting Help topic.

At line:1 char:1

+ New-PSSession -ConnectionURI "$connectionUri" -ConfigurationName Microsoft.Excha ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin

gTransportException

+ FullyQualifiedErrorId : -2144108173,PSSessionOpenFailed

 

EMS Error #2:

failed with the following error message: [ClientAccessServer=,BackEndServer=,RequestId=d89f6ed1-19

8b-4a00-9941-a08f1bb78c54,TimeStamp=11/5/2018 6:58:02 AM]

[AuthZRequestId=b2c5b4c3-edd9-410c-a7fe-af0499783858][FailureCategory=AuthZ-SetupVersionInformationCorruptException] Unable to determine the installed file version from the registry key 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellEngine'. For more information, see the about_Remote_Troubleshooting Help topic.

At line:1 char:1

 

EAC Error #1 (occurs with EMS Error #2):

Connecting to remote server failed with the following error message : IIS 8.0 Detailed Error - 500.0 - Internal Server Error

 

Cause:

EMS Error #1 occurs because the WS-Management (WSMan) module is not registered / loaded in IIS.

 

EMS Error #2 and EAC Error #1 occurs because Windows PowerShell version 2.0 is not installed.

 

Resolution:

To resolve EMS Error #1 follow these steps.

  1. Open IIS and navigate to the Default Web Site.
  2. In the control pane, click on Modules.
  3. WSMan should not be found in the enabled modules list.
  4. Click on Configure Native Modules in the action pane.
  5. Verify that WSMan is listed there without a check in the box next to it.
    1. If WSMan is not listed, go to step 10.
  6. Navigate to PowerShell virtual directory under Default Web Site.
  7. Click on Modules in the control pane.
  8. Verify WSMan is not in the list of enabled modules.
  9. Click on Configure Native Modules in the action pane. Verify that WSMan is listed without a check in the box next to it.
  10. If WSMan does not appear in the list, click on the Server's Name in the left navigation pane and then click on Modules in the control pane.
  11. Click on Configure Native Modules in the action pane.
  12. Click on the Register button.
  13. Enter "WSMan" in the Name field and "%windir%\system32\wsmsvc.dll" (both entries without the quotation marks). Click OK button to save and register.
  14. WSMan should now appear in the list of native modules without a check in the box next to it.
  15. Open a Command Prompt (w/ administrator privileges) and execute IISRESET /noforce command.

Error EMS #1 should now be resolved when you retry performing the action.

 

To resolve EMS Error #2 and EAC Error #1, perform the following steps to install Windows PowerShell version 2.0.

  1. Open Server Manager.
  2. Click on Manage and then select Add Roles and Features.
  3. Click next a few times in the wizard until you reach Select Features screen.
  4. Expand Windows PowerShell selection and check the box next to Windows PowerShell 2.0 Engine and then keep clicking next until you see an install button.
  5. Click install and wait until the operation completes.
  6. Reboot the server.

When you launch EMS or EAC, you should be able to get into both tools without any issue at this point.

1 - 10Next
 

 Image Viewer

 
 

 About this blog

 
About this blog
Welcome to Xadean's contribution to the consulting community through sharing his anthology of lesson's learned and technical insights.