One of the more common problems that Administrators face when configuring their SharePoint deployment is to adequately guess, in advance, how many web applications should be assigned to a single application pool. By default, the Central Administration UI in Microsoft Office SharePoint Portal Server 2003 asks you to create a new application pool each time you create a new web application. Should you do this? Should each new web application (virtual server) run inside it's own application pool?
One of the upsides to having each web application run inside it's own application pool is that each web application will enjoy the worker process isolation that each application pool would offer. This feature would give your deployment maximum stability, especially if many of your web applications were running home-grown code that could potentially bring down a WSS-based application. In addition, any security exploits that would bring down a web application would need to be replicated across each application pool in order to be effective in bringing down your entire farm.
However, our focus here is on the memory that application pools consume and on offering some ideas on how to capacity plan your application pool and web application topology. So, while it might be a good idea from a security and stability standpoint to run each web application in it's own application pool, common sense tells us that this may not be a good idea. Let's explore this topic more fully.
Review of the Current Literature
If you look around, you'll find that different authors give us different ideas on how many web applications should be associated with an individual application. Here is a sampling of what I've found (and the numbers are not entirely consistent):
The Security Considerations for Server and Site Configurations Guide, published by Microsoft, states that each application pool will consume between 30-50 MB of RAM and that beyond 20 sites, the process will run out of memory to hold all of the site processes:
This article from Microsoft recommends not having more than 50 (compared to 20 in the previous article) portal sites associated with a single application pool. In addition, this article states that one portal with it's own application pool will use roughly 150MB of RAM and that each additional portal site will use around 30-50 MB of RAM. So, in theory, 50 portals would use 2.65 GB of RAM – nearly all of the RAM that would be available in user mode if the administrator booted the machine using the /3GB switch. Hence, if each application pool were given 1GB of RAM, you could potentially place 18 portal servers in a single application pool (1×150 and 17×50 = 1GB RAM) running a single process. In this case, if the server were booted using the 3GB switch, you could potentially host 54 portals on a single server:
This blog exchange would indicate that an application pool can have a foot print as small as 5MB, though one must wonder if the application pool is doing little more than sitting by the pool, drinking an iced tea. Note that both the questioner and the Microsoft person find that 50MB for an app pool is "expensive", while the first article felt that number was within the range of reasonable (an argument from silence, to be sure, but still a logical deduction):
Note this blog entry states that a worker process is limited to 2GB of RAM, making the idea of 50 portal sites in a single application pool impossible, as was suggested in the first article in the blog. Also, this person experienced "memory problems" when the app pool got to using over 1.5GB of RAM. We can only speculate what his problems are, but this is one story of a guy who wouldn't run an app pool over 1.5GB of RAM – note he recycled the app pool automatically at this point.
This blog entry notes that a new portal in a new application pool requires 5x-10x more RAM than adding the same portal to an existing application pool. If a new application pool with a new portal consumes 150GB of RAM, and a second portal uses 30-50 GB of RAM (as documented in other articles above), then the 5X rule would seem to apply IF you use the 30MB of RAM per portal. However, the other end of the spectrum is that if we use the 50MB per portal and multiple that by 10X, then we'd end up with 500MB that a new application pools would have to use to host its' first portal. Not likely, if you ask me………….perhaps a bit of exaggeration:
Several of the other articles I found simply quoted or referenced the articles that you see here. This isn't to say that other articles aren't out there to consider (and you're welcome to post them in your response if you'd like), it's just that I didn't find them quickly.
Architecture Discussion – Skip if you don't like Operating System Architecture Details
Now, a worker process in an application pool is really a single thread (well, I'll talk more about this in a moment) that the subsystem uses to process all of the pages in the web applications. A thread is an entity within a process that Windows Schedules for execution. It is the smallest unit of code that can be run by itself. A thread includes the contents of a set of CPU registers representing the state of the processor, two stacks (one for the the to use while executing in kernel mode and one for executing in user mode), a private storage area for use by subsystems, runtime libraries, and DLLs; and a unique identifier called a thread ID (quoted from Microsoft Windows Internals 4th Edition). These elements, when combined, are called the thread's context.
In an ideal world, each thread would have it's own processor exclusively for it's own use. But we don't live in an ideal world, so we have to share threads between processors. Multitasking is the operating system's way of sharing a single processor between multiple threads of execution. If the computer has more than one processor, it can process as many threads simultaneously as it has processors. The Windows operating system, by default, can run all threads across all processors, meaning that one processor is not dedicated to running one or more threads exclusively. This is known as Symmetric Multiprocessing (SMP). However Windows can be configured to assign one or more threads to run on a specified processor. This is known as Asymmetric Multiprocessing (ASMP). In either case, the memory space for the threads that are run is shared across all of the processors, so having multiple processors does not increase the amount RAM (in a virtual sense) on your system.
It is important to understand the difference between User Mode (UM) and Kernel Mode (KM) in order to understand how memory is mapped. These two modes are setup primarily to protect the operating system components from being damaged or hindered by applications that run on top of the operating system. User application code runs in user mode, whereas operating system code runs in kernel mode. UM apps that call for system services must run through a subsystem .DLL to make their calls for resources. The KM processes retrieve the resources for the UM application. If the UM application needs to manipulate the hardware, such as repainting the screen, the call is proxied to the hardware by KM processes. At no time do UM applications talk directly to the hardware, whereas KM process can and do manipulate hardware directly. KM is a mode of execution in a processor that grants access to all system memory and all CPU instructions. These threads of execution have a higher processing priority than threads that run in UM. KM operating system and device driver code have complete access to system space memory and can bypass Windows security to access objects.
Now, a program is a static sequence of code instructions. A process is a container for a set of resources that are used when executing an instance of the program. A process can be run in either UM or KM, but will contain all of the supporting environment structures that a thread needs to run (see above). Other supporting environment elements that need to be in place when a process runs a thread include items like communication ports, files and semaphores (a semaphore is essentially a counter that allows a maximum number of threads to access the resources of the process that are protected by the semaphore). A process also contains a set of virtual memory address that the process can use, a process ID (seen in Task Manager) and at least one thread of execution.
It is important to understand that every thread within a process shares the process's virtual address space, meaning that all of the threads in a process can write to and from each other's memory. The overall size of the virtual address space varies for each hardware platform, but for 32-bit, X86 systems, the maximum is 4GB. By default, Windows allocates half of this address space to UM processes and the other half to KM processes.
If you have 4GB of RAM, you can boot the server (Windows 2000 Server with SP2 and later) with the /3GB switch and the /USERVA switch. The net effect of using the /3GB switch is that 3GB of the memory address spaces are reserved for UM processes, leaving 1GB for KM processes. The advantage of using the /USERVA switch is that you can specify a value between 2048 and 3072 and that amount of RAM will be given to UM and the balance left for KM. The /3GB switch must also be used, but you can throttle the UM memory space to be 2.5GB, for example, instead of having to choose between 2GB or 3GB when booting the system. These switches are applied to the boot.ini file. Instructions on how to do this can be found here: http://www.winguides.com/registry/display.php/1268/ and here: http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
One of the attractive things about running in a 64-bit environment is that the overall address space for processes increases from 4GB to 7152GB. If you're not running in a 64-bit environment (and more SharePoint servers are not these days), then Windows Server provides a way to map the 32-bit memory spaces to a 64-bit virtual environment. This feature is called Address Windowing Extension and it allows a 32-bit application to allocate up to 64GB of physical memory and then map views or windows, into its 2-GB address space. note that IIS 6.0 has a Large Memory Support feature that enables you to cache up to 64-GB of memory on a traditional x86 system (http://www.microsoft.com/windowsserver2003/evaluation/overview/dotnet/bestplatform.mspx#EHE).
I sometimes get the question if SharePoint can run in a 64-bit environment and the answer is "yes" if the machine is running in 32-bit emulation mode: http://blog.u2u.info/DottextWeb/patrick/archive/2005/10/18/10192.aspx
An application pool will have, by default, one process assigned to run in UM. You can increase the number of processes that will be run and configure a web garden. A web garden is 2-n number of processes that are running in the same application pool. But don't let the word "pool" fool you: each process will get it's own set of supporting elements that it needs to be a complete, independent process, including it's own memory space. See http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/26d8cee3-ec31-4148-afab-b6e089a0300b.mspx?mfr=true and http://www.thescripts.com/forum/thread54826.html. The main advantage of a web garden is that each process provides independent access to the web application. This means that if one process is either busy (for example, servicing a backup request) or blocked, then the other processes can answer calls and give the clients access to the web application. This increases the stability of the web application but it also uses more memory since each process gets its' own memory address space as part of it's context.
So, now that we've gone through a bit of operating system architecture, let's come back to the question of how many web applications can we associate with a single application pool.
First, you need to do some performance monitoring of the process that runs your SharePoint threads. This will be the W3WP processes. You need to *know* how much memory these processes are consuming on average. There will always be spikes and valleys, but the overall average is what you're after.
Secondly, you need to decide if you're going to run the /3GB switch and perhaps the /USERVA switch to allocate more of the 4GB address space to UM, which is where SharePoint runs.
Thirdly, if you've never run SharePoint and are trying to predict the future without any past history, then your best practice here is to guess based on the numbers that Microsoft has provided. What do I think? Well, here's my take on all this, in a nutshell:
Do not run more than 10 virtual servers per application pool until you have some performance monitoring experience behind you and you can point to your own numbers which tell you that your app pools can handle more virtual servers
Do not plan to run more than 20 virtual servers on a single physical server without adding a second (or more) web server in your farm. The reason I say this is because if you're running 20 virtual servers, you've got a growing, busy farm and chances are good that your best practice will be to scale out before you scale up.
Always purchase servers with 4GB of RAM to give yourself maximum flexibility in memory allocation for your application pools. BTW, I also recommend dual proc whenever you can get it. Most are doing this these days, so this is nothing profound.
I recommend running a web garden of 2-3 for each application pool. When stsadm runs, it locks one of the threads for it's own use. Having other threads available to service calls during backup/restore operations is necessary in most environments. In addition, if a thread becomes very busy, the other threads can pick up the slack.
Finally, here's a handy article on 10 steps to improving IIS performance: http://www.remarc.co.uk/remarcnews/Jan06/iisimprovements.aspx
So there you have it. I'd be interested to hear what you all have to say about this post and if you have any additional thoughts on memory allocation to an application pool.
You can find this article here.