This code works (I can hear the relays clicking, and see 0x26 on the i2c scan) (note custom component module4relay is basically unit4relay adapter for a different register to enable the relays)

esphome:
  name: relay-test-s3
  friendly_name: relay-test-s3


esp32:
  board: esp32-s3-devkitc-1
  framework:
    type: arduino

# Enable logging
logger:

# Enable Home Assistant API
api:
  encryption:
    key: "Kh5g1ThAQ3SUqKw9GnyJUI5PyOkndNl4+ykbxHoUIjc="

ota:
  - platform: esphome
    password: "5f65f9ff9e4ab8b3c928a7216c551525"

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  # Enable fallback hotspot (captive portal) in case wifi connection fails
  ap:
    ssid: "Relay-Test Fallback Hotspot"
    password: "G1ypogSTYlkb"

captive_portal:

i2c:
  - id: bus_internal 
    sda: GPIO12
    scl: GPIO11
    scan: true 

  - id: bus_port_a 
    sda: GPIO2
    scl: GPIO1
    scan: true 

external_components:
  - source: github://ngolf/esphome-yaml/components
    components: module4relay

module4relay:
  i2c_id: bus_internal

switch:
  - platform: module4relay
    relay_1:
      name: "Module Relay Channel 1"
      restore_mode: RESTORE_DEFAULT_OFF
    relay_2:
      name: "Module Relay Channel 2"
      restore_mode: RESTORE_DEFAULT_OFF
    relay_3:
      name: "Module Relay Channel 3"
      restore_mode: RESTORE_DEFAULT_OFF
    relay_4:
      name: "Module Channel 4"
      restore_mode: RESTORE_DEFAULT_OFF

But adding the axp2101 component (which I need for the backlight) makes the relay stopping being found on i2c scan, and relays inoperable.

(external_components:)
  - source: github://ngolf/esphome-yaml/components
    components: module4relay

axp2101:
  id: axp2101_pmu
  i2c_id: bus_internal