You can control who can logon to the TS by configuring permissions to RDP in it’s properties in TS configuration and remove the general groups and replace with a group that contains users that you want to be able to have access. To beef up network security look at using ipsec using ESP and having an ipsec require policy on the TS for at least port 3389 TCP. Ipsec also allows you to configure IP addresses in the filter though if not a subnet you need to list IP addresses individually.
The clients you want to access the TS would need to have a compatible ipsec policy such as client/respond. As far as mac addresses you could look into using a managed switch that lets you build a list of allowed mac addresses though ipsec already gives you a huge extra measure of security requiring computer authentication before communications can begin.
Ipsec is a somewhat complex topic and ipsec policies require special considerations/exemptions for domain controllers in that they can not use ipsec to secure any traffic used for authentication between themselves and domain members and testing of an ipsec policy before implementing. The links below explain more on ipsec.