Set up VLESS over mKCP with TLS in Xray

mKCP is an Xray transport that sends traffic over UDP with its own retransmission; the Xray docs say it "sacrifices bandwidth to reduce latency" and "generally consumes more traffic than TCP". This guide sets up VLESS over mKCP with TLS on a VPS set up as in the first ten minutes on a new VPS, using the stable Xray release and a free certificate.

1. Point a domain at the server

TLS needs a certificate, and a certificate needs a domain. Create an A record for the server's IPv4 address and an AAAA record for its IPv6 address, for example vpn.example.com. Both work for clients.

2. Install Xray

$ sudo bash -c "$(curl -L https://github.com/XTLS/Xray-install/raw/main/install-release.sh)" @ install
$ xray uuid
$ openssl rand -hex 16

The UUID identifies the client. The random hex is a password for the packet mask in step 5. This guide was tested with Xray v26.3.27, the stable release the script installs.

3. Open the ports

Certbot needs TCP port 80 to prove you own the domain, now and at every renewal. mKCP runs on UDP port 443. If you run neither ufw nor firewalld, skip this step.

$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/udp

$ sudo firewall-cmd --get-active-zones
$ zone=public
$ sudo firewall-cmd --permanent --zone="$zone" --add-port=80/tcp
$ sudo firewall-cmd --permanent --zone="$zone" --add-port=443/udp
$ sudo firewall-cmd --reload

For firewalld, replace public with the zone the first command shows.

4. Get a certificate

Install certbot. On RHEL-based systems it comes from EPEL, and its renewal timer is off until you turn it on.

$ sudo apt install -y certbot

$ sudo dnf install -y epel-release
$ sudo dnf config-manager --set-enabled crb
$ sudo dnf install -y certbot
$ sudo systemctl enable --now certbot-renew.timer

Xray runs as the user nobody, which cannot read certbot's private key. This script copies the certificate to a place Xray can read and restarts Xray:

#!/bin/sh
set -e
dir=/usr/local/etc/xray/certs
install -d -m 0755 "$dir"
install -m 0644 "$RENEWED_LINEAGE/fullchain.pem" "$dir/fullchain.pem"
install -m 0600 -o nobody "$RENEWED_LINEAGE/privkey.pem" "$dir/privkey.pem"
systemctl restart xray

Make it executable, then request the certificate. Certbot runs the script now and again after every renewal.

$ sudo chmod 755 /usr/local/sbin/xray-cert-hook
$ sudo certbot certonly --standalone -d vpn.example.com --deploy-hook /usr/local/sbin/xray-cert-hook

5. Write the server config

{
  "log": { "loglevel": "warning" },
  "inbounds": [
    {
      "listen": "0.0.0.0",
      "port": 443,
      "protocol": "vless",
      "settings": {
        "clients": [ { "id": "YOUR-UUID" } ],
        "decryption": "none"
      },
      "streamSettings": {
        "network": "kcp",
        "security": "tls",
        "tlsSettings": {
          "certificates": [
            {
              "certificateFile": "/usr/local/etc/xray/certs/fullchain.pem",
              "keyFile": "/usr/local/etc/xray/certs/privkey.pem"
            }
          ]
        },
        "finalmask": {
          "udp": [
            { "type": "salamander", "settings": { "password": "YOUR-PASSWORD" } }
          ]
        }
      }
    }
  ],
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [ { "ip": ["geoip:private"], "outboundTag": "block" } ]
  },
  "outbounds": [
    { "protocol": "freedom", "tag": "direct" },
    { "protocol": "blackhole", "tag": "block" }
  ]
}

Older guides put header and seed inside kcpSettings. Current Xray removed both and refuses to start with them. The finalmask block replaces them: Salamander, a method taken from Hysteria2, scrambles each UDP packet with your password, so the server stays silent to anything without it. Some guides use the mkcp-aes128gcm mask instead. The Xray core inside v2rayNG 2.2.6 does not know that name and stops with unknown config id: mkcp-aes128gcm.

Leave flow out, because Vision only works over TCP. The routing block stops clients from reaching services on the server itself or its private network.

$ sudo xray run -test -config /usr/local/etc/xray/config.json
$ sudo systemctl restart xray
$ sudo ss -lnup | grep ':443'

The last command must show Xray on UDP 443. The current Xray docs write "method": "mkcp", but that field only exists in pre-release builds. On the stable release, a config with it still passes the test, then quietly runs on plain TCP.

6. Connect a client

Build the link from your values. The fm part is the finalmask block, URL-encoded; only the password changes.

vless://YOUR-UUID@vpn.example.com:443?encryption=none&security=tls&sni=vpn.example.com&fp=chrome&type=kcp&fm=%7B%22udp%22%3A%5B%7B%22type%22%3A%22salamander%22%2C%22settings%22%3A%7B%22password%22%3A%22YOUR-PASSWORD%22%7D%7D%5D%7D#MyVPS

In v2rayNG, copy the link, tap +, then Import from Clipboard. Tap the new entry to select it, then tap the connect button and allow the VPN request. Other clients need the same network, TLS serverName and finalmask as the server.

7. When it does not connect

• If the client waits and then times out, the mask password differs, or UDP 443 is blocked by a firewall.
• If the TLS handshake fails at once, the domain in the link does not match the certificate.
• If ss -lnup shows nothing on 443, read sudo journalctl -u xray -n 50.

Quick reference

Where to go next

mKCP uses more bandwidth than TCP, so it works best as a second endpoint next to VLESS and REALITY. REALITY needs TCP 443 and mKCP uses UDP 443, so both can run on one server.

If you also run REALITY on this server, turn on BBR. It speeds up REALITY's TCP and does not change mKCP, which uses UDP:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

$ sudo sysctl --system
$ sysctl net.ipv4.tcp_congestion_control

Before you order, use the Looking Glass to test the route from your network to each location.