An open system, in the context of computing, is a computer system that combines portability and interoperability, and makes use of open software standards. It typically refers to a computer system that is interoperable between different vendors and standards, allowing for modularity so that hardware and software need not be attached to a single vendor or platform.
Before the popularity of the Windows OS and the PC, open systems used to refer to computer systems with Unix-like operating systems that accepted any modules or programs from any third party that used the same standard, as opposed to that era's closed systems such as IBM computers.
An open system typically refers to a computer system that is interoperable between different software and hardware that adhere to the same open standard. In this case, open means almost anyone can use the standard without legal or financial concern, as opposed to proprietary systems, which have legal agreements usually involving mode of use and royalty fees. For most, this would apply to Linux and other Unix-like operating systems and software.
However, the popularity of Windows and the PC means that third-party software and hardware is often interoperable with these systems, and most software made for Windows is portable to any other computer system running other versions of Windows. Because of this interoperability and portability, and despite Windows being a proprietary OS, a computer running the Windows OS can be considered as an open system.