X server is a server program that connects X terminals running on the X Window System, whether locally or in a distributed network. The X server is installed with the X Window System, which is a cross-platform and complete client-server system for managing graphical user interfaces on a single computer or networked ones. The X server manages the X clients and does the actual work in terms of managing input and display devices and performing requested operations. This simplifies programming, as the application programs themselves do not need to be aware of the hardware details and just rely entirely on the X server.
Techopedia explains X Server
The X server manages X clients, but the relationship is reversed compared to traditional client-server model applications. Each local machine contains the X server, and the X clients are run on the remote machines, but may also run in the same local machine as the X server.
In traditional client-server implementations, the user of the client requests data from the server, which then displays them on the user's screen through the client. In the case of the X system, however, the user controls the server in order to control clients residing in remote workstations so that multiple clients can be controlled at once, providing the user with different applications that are running in different machines. In this way, more tasks can be done without slowing down the user's machine.
The X server provides the following basic types of services: