🤖Have you ever tried Chat.M5Stack.com before asking??😎
    M5Stack Community
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Register
    • Login

    How to debug Module LLM Kit when it silently dies?

    Scheduled Pinned Locked Moved SOFTWARE
    llm
    4 Posts 1 Posters 1.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      riywo
      last edited by

      Hi, I've just got Module LLM Kit and got started with CoreS3 Lite by running some examples.

      TextAssistant works well:
      https://github.com/m5stack/M5Module-LLM/blob/main/examples/TextAssistant/TextAssistant.ino

      However, TTS never works. It keeps sending the text to TTS module but no audio sounds:
      https://github.com/m5stack/M5Module-LLM/blob/main/examples/TTS/TTS.ino

      I installed many apt packages but nothing changed. So, I tried strace because I don't know where the log output is and realized that llm-sys restarted once it receives a message. I couldn't spot the exact message yet but the last message was something like this:

      [pid  3285] write(1, "W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\n\33[1;30;33m[W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys\33[0m\nllm_sys will be exit!\n", 1526) = 1526

      Q: Is there anyway to debug more on the LLM Kit side?

      Update:

      I realized they are systemd services. So, these are the logs:

      llm-tts.service

      root@m5stack-LLM:~# journalctl -u llm-tts -f
Oct 01 03:34:11 m5stack-LLM systemd[1]: Started llm-tts Service.
Oct 01 03:34:45 m5stack-LLM llm_tts-1.6[10412]: [I][operator()][ 189]: serial_zmq_url:ipc:///tmp/llm/5556.sock
Oct 01 03:34:45 m5stack-LLM llm_tts-1.6[10412]: [W][__sigint][  24]: llm_sys will be exit!
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopping llm-tts Service...
Oct 01 03:34:45 m5stack-LLM systemd[1]: llm-tts.service: Deactivated successfully.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopped llm-tts Service.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Started llm-tts Service.

      llm-sys.service

      Oct 01 03:34:10 m5stack-LLM systemd[1]: Started llm-sys Service.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopping llm-sys Service...
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: server listen on port 10001, listenfd=18 ...
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: [W][zmq_bus_publisher_push][ 171]: zmq_bus_publisher_push failed, not have work_id:sys
Oct 01 03:34:45 m5stack-LLM llm_sys-1.6[10266]: llm_sys will be exit!
Oct 01 03:34:45 m5stack-LLM systemd[1]: llm-sys.service: Deactivated successfully.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Stopped llm-sys Service.
Oct 01 03:34:45 m5stack-LLM systemd[1]: Started llm-sys Service.

      My environment:

      • M5Stack CoreS3 Lite
        • PlatformIO
          • pioarduino stable
          • lib_deps = M5Unified, M5Module-LLM
      • Module LLM Kit
        • apt update and apt upgrade
        • apt list | grep llm | grep installed is below:
      lib-llm/stable,now 1.8 arm64 [installed]
llm-asr/stable,now 1.7 arm64 [installed]
llm-audio-en-us/now 0.2 arm64 [installed,local]
llm-audio-zh-cn/now 0.2 arm64 [installed,local]
llm-audio/stable,now 1.6 arm64 [installed]
llm-camera/stable,now 1.9 arm64 [installed]
llm-kws/stable,now 1.9 arm64 [installed]
llm-llm/stable,now 1.9 arm64 [installed]
llm-melotts-zh-cn/now 0.2 arm64 [installed,local]
llm-melotts/stable,now 1.9 arm64 [installed]
llm-model-melotts-en-default/stable,now 0.6 arm64 [installed]
llm-model-melotts-en-us/stable,now 0.6 arm64 [installed]
llm-model-melotts-ja-jp/stable,now 0.6 arm64 [installed]
llm-model-qwen2.5-0.5b-prefill-20e/stable,now 0.2 arm64 [installed]
llm-model-qwen3-0.6b-ax630c/stable,now 0.4 arm64 [installed]
llm-model-sherpa-onnx-kws-zipformer-gigaspeech-3.3m-2024-01-01/stable,now 0.3 arm64 [installed]
llm-model-silero-vad/stable,now 0.4 arm64 [installed]
llm-model-smolvlm-500m-ax630c/stable,now 0.4 arm64 [installed]
llm-model-whisper-tiny/stable,now 0.4 arm64 [installed]
llm-openai-api/stable,now 1.7 arm64 [installed]
llm-sherpa-ncnn-streaming-zipformer-20m-2023-02-17/now 0.2 arm64 [installed,local]
llm-sherpa-ncnn-streaming-zipformer-zh-14m-2023-02-23/now 0.2 arm64 [installed,local]
llm-sherpa-onnx-kws-zipformer-wenetspeech-3.3m-2024-01-01/now 0.2 arm64 [installed,local]
llm-single-speaker-english-fast/now 0.2 arm64 [installed,local]
llm-single-speaker-fast/now 0.2 arm64 [installed,local]
llm-skel/stable,now 1.5 arm64 [installed]
llm-sys/stable,now 1.6 arm64 [installed]
llm-tts/stable,now 1.6 arm64 [installed]
llm-vad/stable,now 1.8 arm64 [installed]
llm-vlm/stable,now 1.9 arm64 [installed]
llm-whisper/stable,now 1.8 arm64 [installed]
llm-yolo11n-pose/now 0.2 arm64 [installed,local]
llm-yolo11n-seg/now 0.2 arm64 [installed,local]
llm-yolo11n/now 0.2 arm64 [installed,local]
llm-yolo/stable,now 1.9 arm64 [installed]
      1 Reply Last reply Reply Quote 0
      • R Offline
        riywo
        last edited by

        Update 2:

        I confirmed TTS itself is working if it is used through OpenAI API server: https://docs.m5stack.com/en/stackflow/openai_api/text_to_speech

        1 Reply Last reply Reply Quote 0
        • R Offline
          riywo
          last edited by

          Update 3:

          Captured this log sometime:

          Oct 01 11:18:51 m5stack-LLM systemd[1]: llm-tts.service: Main process exited, code=killed, status=11/SEGV
Oct 01 11:18:51 m5stack-LLM systemd[1]: llm-tts.service: Failed with result 'signal'.
Oct 01 11:18:52 m5stack-LLM systemd[1]: llm-tts.service: Scheduled restart job, restart counter is at 1.
Oct 01 11:18:52 m5stack-LLM systemd[1]: Stopped llm-tts Service.
Oct 01 11:18:52 m5stack-LLM systemd[1]: Started llm-tts Service.
          1 Reply Last reply Reply Quote 0
          • R Offline
            riywo
            last edited by

            Ok, it worked finally. At least, I needed to use module_llm.melotts instead of module_llm.tts.

            The model specification in tts_config doesn't matter for my case. It picks melotts-en-default somehow automatically even though the default value is melotts_zh-cn.

            The module is very unstable when the controller is powered on. It requires several clicks of the reset button until it finally speaks. (The return value of melotts.inference is -97 when it's successful.)

            1 Reply Last reply Reply Quote 0

            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

            With your input, this post could be even better 💗

            Register Login
            • First post
              Last post