Nick's Tech Blog

It is not immediately obvious how to install an existing digital certificate to HP’s Intelligent Management Center (iMC). Thankfully, it is possible although via a somewhat convoluted process.

The information needed to do this is derived from an article on HP’s Support Center:

HP iMC – How to install User Certificates on the iMC Server

This documentation is clearly left wanting as it only explains how to create a self-signed digital certificate for iMC using Java’s keytool but not how to import and use an existing one.

From the documentation, we learn that:

• A Java keystore is used to hold the certificate. It is located at “iMC installation path\iMC\client\security\” with a file name of “keystore“.
• The password to the keystore is “iMCV300R002“.
• Within the keystore, a key with alias “imc” is stored and this is protected with the same password of “iMCV300R002“.

Assuming that the existing digital certificate is in PFX / PKCS12 format and includes a complete chain to the root:

1) Stop the iMC service.

2) At the command line, from Java’s bin directory and substituting paths as appropriate, convert the certificate to the required format:

keytool -importkeystore -srckeystore c:\certificate.pfx -destkeystore c:\keystore -srcstoretype pkcs12 -deststoretype JKS -storepass iMCV300R002 -v

You will be asked for the existing password to the source keystore and will be given the entry alias if the import is successful. Take a note of it.

Enter source keystore password:
Entry for alias le-5aa48bcf-bb89-4447-bdc5-1a1aa868355c successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or
cancelled
[Storing c:\keystore]

3) Clone the imported key using the noted alias, naming the cloned key “imc”:

keytool -keyclone -keystore c:\keystore -alias le-5aa48bcf-bb89-4447-bdc5-1a1aa868355c -dest imc

You will be asked for the password to the keystore that has been created, which is “iMCV300R002“, and the password to the imported key, which is the password to the original source keystore.
When asked for the password for the cloned key that is to be created, specify “iMCV300R002” and confirm it.

Enter keystore password:
Enter key password for <le-5aa48bcf-bb89-4447-bdc5-1a1aa868355c>
Enter key password for <imc>
(RETURN if same as for <le-5aa48bcf-bb89-4447-bdc5-1a1aa868355c>)
Re-enter new password:

4) Delete the now superfluous key from the keystore by specifying its alias and giving the keystore password, “iMCV300R002“:

keytool -delete -keystore c:\keystore -alias le-5aa48bcf-bb89-4447-bdc5-1a1aa868355c
Enter keystore password:

5) Replace the existing keystore file in “iMC installation path\iMC\client\security\” with the one that has been created.

6) Start the iMC service.

It is a seldom known fact that, as of the NT branch, Windows supports case sensitivity. Using it, however, is non-obvious, requiring knowledge of the design and inner workings of the operating system.

Beneath the Win32 environment subsystem that most programmers are familiar with lies the NT Executive. It is accessed via the NT Native API and, at this level, programmers must be familiar with the OBJECT_ATTRIBUTES structure that is used when creating or opening executive objects. It is a knowledge of this structure, and how the Win32 subsystem uses it, that is required to truly understand case handling in Windows.

Examples of such objects are Files, Registry Keys, I/O Completion Ports, Events, Jobs, KeyedEvents, Mutants (Mutexs), Object Directories, Processes, Sections, Semaphores, Symbolic Links, Threads and Timers.

When it comes to case sensitivity, however, most users will be concerned with only one object type, Files.

The OBJECT_ATTRIBUTES structure, as documented, “specifies attributes that can be applied to objects or object handles by routines that create objects and/or return handles to objects“.

One of its members is the Attributes bitmask that specifies the attributes for the handle to an object.

The flag within this that is of interest is the OBJ_CASE_INSENSITIVE flag.
It is documented as follows:

If this flag is specified, a case-insensitive comparison is used when matching the name pointed to by the ObjectName member against the names of existing objects. Otherwise, object names are compared using the default system settings.

This means that when the flag is specified, case insensitive operation is guaranteed, however, when it is not, it is left to the system to decide.

This documentation is arguably incomplete as it does not state what the ‘default system settings’ are, or how to control them.

It turns out that there are two parts to this. Firstly, a single setting that controls case sensitivity on system-wide basis, obcaseinsensitive, and then, assuming that case sensitive operation is supported, file system support on a per-volume basis.

The obcaseinsensitive value is located in the registry at “\REGISTRY\MACHINE\System\CurrentControlSet\Control\Session Manager\Kernel” and is a DWORD. It is read only as a system boots.
(In Win32 nomenclature, the path is “HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Kernel”.)

• When set to 0, the object manager runs in case sensitive mode.
• When set to 1, the object manager runs in case insensitive mode.
• When unspecified, NT 5.1 (Windows XP) and later editions default to running in case insensitive mode.
• obcaseinsensitive has no meaning in NT 5.0 (Windows 2000) and prior versions of NT, which always run in case sensitive mode.

Case insensitive mode is only implemented for certain object types and not for all. These are Object Directories (NtCreateDirectoryObject, NtOpenDirectoryObject), Symbolic Links (NtCreateSymbolicLinkObject and NtOpenSymbolicLinkObject) and I/O objects, such as Files (NtCreateFile, NtOpenFile) and I/O Completion Ports (NtCreateIoCompletion, NtOpenIoCompletion).

At the Win32 environment subsystem, the only exposure to case sensitivity is for File objects and via the following APIs:

• CreateFile / CreateFileTransacted and the FILE_FLAG_POSIX_SEMANTICS flag, which ensures that the OBJ_CASE_INSENSITIVE flag is not set in the OBJECT_ATTRIBUTES structure used in internal calls to NtCreateFile.
• FindFirstFileEx / FindFirstFileTransacted and the FIND_FIRST_EX_CASE_SENSITIVE flag, which ensures that the OBJ_CASE_INSENSITIVE flag is not set in the OBJECT_ATTRIBUTES structure used in internal calls to NtCreateFile to open a directory file before subsequent calls to NtQueryDirectoryFile.

(It is worth noting that most existing Win32 applications will not set the FILE_FLAG_POSIX_SEMANTICS flag or the FIND_FIRST_EX_CASE_SENSITIVE flag when calling these APIs, or will rely on other APIs that internally do not. Because of this, the OBJ_CASE_INSENSITIVE flag will always end up being set by the Win32 subsystem in the OBJECT_ATTRIBUTES structures that it creates to call NT Native APIs. This always enforces insensitivity for such applications.)

Official documentation from Microsoft for obcaseinsensitive is sparse and patchy:

• Configure Case Sensitivity for File and Folder Names
• Semantic Differences Between Server for NFS and UNIX-Based NFS Servers
• Enable case sensitive behavior with Windows XP and Interix Subsystem or SFU

The value is also managed by group policy, where the best explanation given by Microsoft can also be found:

This security setting determines whether case insensitivity is enforced for all subsystems. The Win32 subsystem is case insensitive. However, the kernel supports case sensitivity for other subsystems, such as POSIX.

If this setting is enabled, case insensitivity is enforced for all directory objects, symbolic links, and IO objects, including file objects. Disabling this setting does not allow the Win32 subsystem to become case sensitive.

• System objects: Require case insensitivity for non-Windows subsystems

There are times that applications need to determine if a Windows system is running with case insensitivity, perhaps for path comparison. A naive implementation will simply query the obcaseinsensitive value from the registry, but this has drawbacks:

1. The value only represents how the NT executive’s object manager will operate at the next boot, not necessarily how the system is right now.
2. NT 5.0 (Windows 2000) and prior, if supported, must be special cased as case insensitivity is not supported. Further, the obcaseinsensitive value may be present in the registry even though it is meaningless.
3. Future versions of Windows may decide to implement things differently, such as in a more granular way. (Perhaps on a per-thread or per-process basis via a privilege.)

The solution is to use NtOpenSymbolicLinkObject to attempt to open the known symbolic link, “SystemRoot”, which resides in the root directory of the object namespace with incorrect casing.

By attempting to open “\SYSTEMROOT” with a DesiredAccess of 0×0, which means no access rights to the object, and with no flags specified in the attributes member of the OBJECT_ATTRIBUTES structure passed, the current operational mode can be easily and efficiently determined.

• Where the object manager is running in case insensitive mode, a NTSTATUS value of STATUS_ACCESS_DENIED will be returned.
• Where the object manager is running in case sensitive mode, a NTSTATUS value of STATUS_OBJECT_NAME_NOT_FOUND will be returned.

(If the SYMBOLIC_LINK_QUERY specific access right (0×1) is requested, the open operation will succeed if the user is permitted to open the symbolic link via its DACL. This is the case typically only if the user is an Administrator. The operation will fail and STATUS_ACCESS_DENIED will be returned otherwise.)

Redhat’s Corinna Vinschen of Cygwin fame pointed out that the same check can also be achieved via NtOpenDirectoryObject in much the same way.

• Where the object manager is running in case insensitive mode, a NTSTATUS value of STATUS_OBJECT_TYPE_MISMATCH will be returned.
• Where the object manager is running in case sensitive mode, a NTSTATUS value of STATUS_OBJECT_NAME_NOT_FOUND will be returned.

(Cygwin has had a patch committed to its source code to ensure that it now tests for case sensitivity rather than reading the obcaseinsensitive value from the registry.)

As mentioned previously, once support for case sensitivity system-wide has been established, the ability to use files in a case sensitive way is then determined by the file system in use on a per-volume basis.

This is established by the presence of the FILE_CASE_SENSITIVE_SEARCH file system attribute flag for a volume, documented as meaning that “the specified volume supports case-sensitive file names”.

File system attributes for a volume are queried via either:

• The NT Native API by NtQueryVolumeInformationFile, requesting the FileFsAttributeInformation information class and looking at the FileSystemAttributes member of the FILE_FS_ATTRIBUTE_INFORMATION structure.
• The Win32 API by GetVolumeInformation or GetVolumeInformationByHandle and the lpFileSystemFlags out parameter.

I read with interest the posting on Microsoft’s new file system, ReFS, over on Microsoft’s Building Windows 8 blog:

Building the next generation file system for Windows: ReFS

For programmers who work with any of the interesting features of NTFS and have not read it yet, they should find the time to do so. It is, potentially, just about to disrupt their world.

One of the goals for the file system is apparently to:

Maintain a high degree of compatibility with a subset of NTFS features that are widely adopted while deprecating others that provide limited value at the cost of system complexity and footprint.

And snuck away in the FAQ section is the following:

Q) What semantics or features of NTFS are no longer supported on ReFS?

The NTFS features we have chosen to not support in ReFS are: named streams, object IDs, short names, compression, file level encryption (EFS), user data transactions, sparse, hard-links, extended attributes, and quotas.

Mapped to File System Attribute Flags, the following are not going to be supported:

• FILE_NAMED_STREAMS
  The file system supports named streams.
• FILE_SUPPORTS_OBJECT_IDS
  The file system supports object identifiers.
• FILE_FILE_COMPRESSION
  The file system supports file-based compression.
• FILE_SUPPORTS_ENCRYPTION
  The file system supports the Encrypted File System (EFS).
• FILE_SUPPORTS_TRANSACTIONS
  The file system supports transactions.
• FILE_SUPPORTS_SPARSE_FILES
  The file system supports sparse files.
• FILE_SUPPORTS_HARD_LINKS
  The file system supports hard links.
• FILE_SUPPORTS_EXTENDED_ATTRIBUTES
  The file system supports extended attributes.
• FILE_VOLUME_QUOTAS
  The file system supports per-user quotas.

This is a large list of features that are not going to work in ReFS, at least in its initial implementation. It has the potential to have significant compatibility implications and to break many of the existing applications that are in use today.

To ensure compatibility of applications going forward, developers should ensure that they properly check that a feature is available before attempting to use it. This comes with a caveat:

The FILE_SUPPORTS_HARD_LINKS, FILE_SUPPORTS_EXTENDED_ATTRIBUTES, FILE_SUPPORTS_OPEN_BY_FILE_ID and FILE_SUPPORTS_USN_JOURNAL flags were only added in NT 6.1 (Windows 7 and Windows Server 2008 R2) and yet support for all of them has been implicit since NTFS 3.0 that was introduced with NT 5.0 (Windows 2000). A literal check for NTFS by name is therefore also needed to determine the implicit presence of these features.

(It is important to also ensure that such literal checks for NTFS by name are only performed to determine the implicit support of these features and for no other purpose. This is to ensure that applications are as forward compatible with other file systems as is reasonably possible by not taking a dependency on NTFS.)

For applications that depend on any of the features that are not available, appropriate workarounds will have to be found or a declaration made that they are not compatible with the underlying file system due to a lack of feature support.

This information is available in the Win32 subsystem via GetVolumeInformation and GetVolumeInformationByHandle, and at the underlying NT layer via NtQueryVolumeInformationFile.

The good news? Many of the features that are missing are relied upon heavily in Windows internally. This explains why it will not be possible to use ReFS as the file system for the boot volume. ReFS appears to be merely in its infancy and it has simply not been implemented fully yet, many of the missing features to appear by the time the file system becomes available in client versions of Windows.

There are a few edge cases when a programmer has no choice but to terminate a hung thread.

An example of this being where an unknown handle originating from another process is being queried via the partially documented NtQueryObject() to establish its type. Prior to the NT 6.1 kernel (Windows 7 and Windows Server 2008 R2), the call would hang indefinitely for pipes and the only way to recover from this was to time out the operation and to forcefully terminate the thread.

Microsoft provide a documented Win32 API to do this, TerminateThread(). This ultimately calls the undocumented NT API, NtTerminateThread() to request termination of a thread.

(The SYNCHRONIZE standard access right is not required to be granted on the thread’s handle for it to be terminated by TerminateThread, but beware, thread termination completes asynchronously. This means that the caller must wait on the thread for it to become signalled to know that termination has completed, for this, the right is required. This behaviour is not documented, but is implicit.)

It is important to note that terminating a thread is generally a terrible idea and should not be carried out in the vast majority of cases. It is usually the wrong thing to do.

When terminating a thread that has been created via the Win32 API, it is important to terminate it via the same means, at the same level of abstraction. This is to allow book keeping and internal cleanup to take place appropriately, fundamentally to not unnecessarily break the abstraction via a layering violation. This is true even where such internal mechanisms are not implemented in the abstraction today, they may be in the future.

Among the many caveats listed of forcefully terminating a thread, one that easily bites is that 1 MiB is, by default, leaked per-thread-terminated under versions of Windows prior to and including Windows Server 2003 (NT 5.2 kernel):

“Windows Server 2003 and Windows XP/2000:  The target thread’s initial stack is not freed, causing a resource leak.”

As of the NT 6.0 kernel (Windows Vista and Windows Server 2008), the thread’s user mode stack is freed when a thread is terminated.

Thankfully, there is an undocumented API that can be used to easily free the thread’s user mode stack in older versions of Windows, RtlFreeUserThreadStack(). It is exported via ntdll.dll.

This function was first exported in the NT 4.0 kernel (Windows NT 4.0) and was discontinued in the NT 6.0 kernel (Windows Vista and Windows Server 2008). This means that, for all the versions of Windows that will realistically be in use today, the ability to easily free a thread’s user mode stack if it must be forcefully terminated is available. Its availability also tallies perfectly with the versions of Windows for which this leak occurs.

Internally, it does the following:

NTSTATUS RtlFreeUserThreadStack(
    HANDLE hProcess,
    HANDLE hThread)
{
    NTSTATUS Status;
    PTEB Teb;
    THREAD_BASIC_INFORMATION ThreadInfo;
    PVOID StackDeallocationBase;
    ULONG Length;
    SIZE_T Size;
    Status = NtQueryInformationThread(hThread,
        ThreadBasicInformation,
        &ThreadInfo,
        sizeof(ThreadInfo),
        NULL);
    Teb = ThreadInfo.TebBaseAddress;
    if (NT_SUCCESS(Status) && Teb)
    {
        Status = NtReadVirtualMemory(hProcess,
            &Teb->DeallocationStack,
            &StackDeallocationBase,
            sizeof(StackDeallocationBase),
            &Length);
        if (NT_SUCCESS(Status) && StackDeallocationBase)
        {
            Size = 0;
            Status = NtFreeVirtualMemory(hProcess,
                &StackDeallocationBase,
                &Size,
                MEM_RELEASE);
        }
    }

    return Status;
}

(Note: The offset to the DeallocationStack member of the 32-bit TEB structure is 0xE0C. The offset to the same member of the 64-bit TEB is 0×1478.)

Before calling the RtlFreeUserThreadStack() API, ensure that the target thread has been suspended to guarantee that it will not be active when its user mode stack is freed. This avoids the potential implications of the race condition that exists when you decide to terminate a thread that is considered hung, it may well spring back to life after the decision to terminate it has been made and before it is terminated.

The API is documented unofficially below:

The RtlFreeUserThreadStack routine frees a thread’s initial stack given a handle to the containing process and handle to the thread where the stack should be freed.

Syntax

NTSTATUS NTAPI RtlFreeUserThreadStack(
__in  HANDLE ProcessHandle,
__in  HANDLE ThreadHandle
)

Parameters

hProcess [in]
Handle to the process in which the thread to be freed resides.

hThread [in]
Handle to the thread that should have its initial stack freed.

Return Value

STATUS_SUCCESS or an appropriate error status.

Remarks

When the decision has been made to terminate a thread:

• Suspend the thread via SuspendThread() to ensure it will not be active when its user mode stack is freed.
• Free the thread’s user mode stack via RtlFreeUserThreadStack() if the API is available.
• Request termination of the thread via TerminateThread().
• Wait for the thread to become signalled via WaitForSingleObject(), indicating that the thread has finished terminating.
  (The SYNCHRONIZE standard access right is required to be granted on the thread handle to wait on the thread.)