🧱

ComfyUI

ComfyUI is an extremely powerful Stable Diffusion workflow builder. It uses a node-based GUI and is for advanced users.

Where to find it in the builder

Sample

You can drag your (API) graph .json in here and run it!

Graph checks

  • ✅ you need to have one SaveImage or VHS_VideoCombine node, the engine will search for this and set it as the last_node_id.
  • ✅all loadImage image fields need to be valid image links (jpg and png preferred). You can get these by uploading an image with the image uploading block. Downloading videos is not yet supported.

Compatible models

├── ultralytics
│ ├── segm
│ │ └── person_yolov8m-seg.pt
│ └── bbox
│ ├── hand_yolov8s.pt
│ └── face_yolov8m.pt
├── sams
│ └── sam_vit_b_01ec64.pth⚠️ -> use "sam_vit_b (375MB)"
├── grounding-dino
│ ├── GroundingDINO_SwinB.cfg.py
│ ├── groundingdino_swint_ogc.pth⚠️ -> use "GroundingDINO_SwinT_OGC (694MB)"
│ ├── groundingdino_swinb_cogcoor.pth⚠️ -> use "GroundingDINO_SwinB (938MB)"
│ └── GroundingDINO_SwinT_OGC.cfg.py
├── embeddings
│ ├── negative_hand-neg.pt
│ ├── easynegative.safetensors
│ ├── bad_prompt_version2-neg.pt
│ └── ng_deepnegative_v1_75t.pt
├── loras
│ ├── theovercomer8sContrastFix_sd15.safetensors
│ ├── more_details.safetensors
│ ├── lcm-lora-sdxl.safetensors
│ ├── lcm-lora-sdv1-5.safetensors
│ ├── sd_xl_offset_example-lora_1.0.safetensors
│ ├── add_detail.safetensors
│ └── theovercomer8sContrastFix_sd21768.safetensors
├── vae
│ ├── sdxl_vae.safetensors
│ ├── kl-f8-anime2.ckpt
│ ├── orangemix.vae.pt
│ └── vae-ft-mse-840000-ema-pruned.safetensors
├── upscale_models
│ ├── RealESRGAN_x4.pth
│ ├── RealESRGAN_x2.pth
│ ├── 4x_foolhardy_Remacri.pth
│ ├── 4x_NMKD-Siax_200k.pth
│ ├── 8x_NMKD-Superscale_150000_G.pth
│ ├── 4x-UltraSharp.pth
│ ├── ESRGAN_4x.pth
│ └── 4x-AnimeSharp.pth
├── ipadapter
│ ├── ip-adapter-plus-face_sd15.safetensors
│ ├── ip-adapter-faceid_sd15.bin
│ ├── ip-adapter-faceid_sdxl.bin
│ ├── ip-adapter-full-face_sd15.safetensors
│ ├── ip-adapter-plus_sd15.safetensors
│ ├── ip-adapter_sd15_light.safetensors
│ ├── ip-adapter-faceid-plus_sd15.bin
│ ├── ip-adapter_sd15_vit-G.safetensors
│ ├── ip-adapter-plus_sdxl_vit-h.safetensors
│ ├── ip-adapter-faceid-plusv2_sd15.bin
│ ├── ip-adapter_sdxl.safetensors
│ ├── ip-adapter-faceid-plusv2_sdxl.bin
│ ├── ip-adapter-plus-face_sdxl_vit-h.safetensors
│ ├── ip-adapter_sdxl_vit-h.safetensors
│ ├── ip-adapter-faceid-portrait_sd15.bin
│ └── ip-adapter_sd15.safetensors
├── controlnet
│ ├── control_v11p_sd15_inpaint_fp16.safetensors
│ ├── depth-zoe-xl-v1.0-controlnet.safetensors
│ ├── control-lora-depth-rank256.safetensors
│ ├── control-lora-canny-rank128.safetensors
│ ├── control_v11e_sd15_shuffle_fp16.safetensors
│ ├── control_boxdepth_LooseControlfp16.safetensors
│ ├── control_sd15_inpaint_depth_hand_fp16.safetensors
│ ├── control_v11p_sd15_openpose_fp16.safetensors
│ ├── control_v11p_sd15_seg_fp16.safetensors
│ ├── control_v11f1e_sd15_tile_fp16.safetensors
│ ├── control_v11p_sd15_mlsd_fp16.safetensors
│ ├── control_v11f1p_sd15_depth_fp16.safetensors
│ ├── control_v11p_sd15_canny_fp16.safetensors
│ ├── controlnet-sd-xl-1.0-softedge-dexined.safetensors
│ ├── control-lora-sketch-rank256.safetensors
│ ├── control-lora-recolor-rank256.safetensors
│ ├── control_v11p_sd15_lineart_fp16.safetensors
│ ├── control-lora-recolor-rank128.safetensors
│ ├── control-lora-canny-rank256.safetensors
│ ├── control_v11p_sd15s2_lineart_anime_fp16.safetensors
│ ├── control_v11p_sd15_normalbae_fp16.safetensors
│ ├── control-lora-depth-rank128.safetensors
│ ├── control-lora-sketch-rank128-metadata.safetensors
│ ├── control_v11u_sd15_tile_fp16.safetensors
│ ├── temporalnetversion2.ckpt
│ ├── control_v11e_sd15_ip2p_fp16.safetensors
│ ├── control_v11p_sd15_scribble_fp16.safetensors
│ ├── OpenPoseXL2.safetensors
│ └── control_v11p_sd15_softedge_fp16.safetensors
├── clip_vision
│ ├── clip_vision_g.safetensors
│ ├── CLIP-ViT-H-14-laion2B-s32B-b79K.safetensors
│ ├── clip-vit-large-patch14.bin
│ └── CLIP-ViT-bigG-14-laion2B-39B-b160k.safetensors
└── checkpoints
├── Realistic_Vision_V5.1_fp16-no-ema.safetensors
├── AOM3A3_orangemixs.safetensors
├── Realistic_Vision_V6.0_NV_B1_fp16.safetensors
├── sd_xl_refiner_1.0.safetensors
├── sd_xl_refiner_1.0_0.9vae.safetensors
├── v1-5-pruned-emaonly.ckpt
├── wd-illusion-fp16.safetensors
├── v2-1_768-ema-pruned.safetensors
├── DreamShaper8_LCM.safetensors
├── albedobaseXL_v12.safetensors
├── AbyssOrangeMix2_hard.safetensors
├── albedobaseXL_v13.safetensors
├── v2-1_512-ema-pruned.safetensors
├── AOM3A1_orangemixs.safetensors
├── sd_xl_base_1.0_0.9vae.safetensors
└── sd_xl_base_1.0.safetensors
​
/root/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/models
├── mm_sd_v15_v2.ckpt
├── v3_sd15_mm.ckpt
├── mm_sd_v15.ckpt
├── mm_sd_v14.ckpt
├── mm_sdxl_v10_beta.ckpt
├── lt_long_mm_16_64_frames.ckpt
├── mm-Stabilized_mid.pth
├── lt_long_mm_16_64_frames_v1.1.ckpt
├── temporaldiff-v1-animatediff.ckpt
├── mm-Stabilized_high.pth
├── lt_long_mm_32_frames.ckpt
└── .gitkeep
​
/root/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/motion_lora
├── .gitkeep
├── v2_lora_ZoomOut.ckpt
├── v2_lora_ZoomIn.ckpt
├── v2_lora_PanLeft.ckpt
├── v2_lora_PanRight.ckpt
├── v2_lora_RollingAnticlockwise.ckpt
├── v2_lora_TiltDown.ckpt
├── v2_lora_RollingClockwise.ckpt
└── v2_lora_TiltUp.ckpt
​

Exporting the graph

  1. 1.
    Enable developer options:
    1. 1.
      Go to settings:\
    2. 2.
      Enable Enable Dev mode Options: \
  2. 2.
    Export the API graph:\

Graph example

This is a graph that should run via our Comfy Block. Mind the cloudinary link for the image field.
expand to show json
{
"3": {
"inputs": {
"seed": 624586032019704,
"steps": 4,
"cfg": 1,
"sampler_name": "lcm",
"scheduler": "normal",
"denoise": 1,
"model": [
"13",
0
],
"positive": [
"6",
0
],
"negative": [
"7",
0
],
"latent_image": [
"5",
0
]
},
"class_type": "KSampler"
},
"4": {
"inputs": {
"ckpt_name": "sd_xl_base_1.0.safetensors"
},
"class_type": "CheckpointLoaderSimple"
},
"5": {
"inputs": {
"width": 1024,
"height": 1024,
"batch_size": 1
},
"class_type": "EmptyLatentImage"
},
"6": {
"inputs": {
"text": "man in space",
"clip": [
"10",
1
]
},
"class_type": "CLIPTextEncode"
},
"7": {
"inputs": {
"text": "text, watermark",
"clip": [
"10",
1
]
},
"class_type": "CLIPTextEncode"
},
"8": {
"inputs": {
"samples": [
"3",
0
],
"vae": [
"4",
2
]
},
"class_type": "VAEDecode"
},
"9": {
"inputs": {
"filename_prefix": "ComfyUI",
"images": [
"8",
0
]
},
"class_type": "SaveImage"
},
"10": {
"inputs": {
"lora_name": "lcm-lora-sdxl.safetensors",
"strength_model": 1,
"strength_clip": 1,
"model": [
"4",
0
],
"clip": [
"4",
1
]
},
"class_type": "LoraLoader"
},
"12": {
"inputs": {
"ipadapter_file": "ip-adapter-plus_sdxl_vit-h.safetensors"
},
"class_type": "IPAdapterModelLoader"
},
"13": {
"inputs": {
"weight": 0.3,
"noise": 0,
"weight_type": "original",
"ipadapter": [
"12",
0
],
"clip_vision": [
"15",
0
],
"image": [
"16",
0
],
"model": [
"10",
0
]
},
"class_type": "IPAdapterApply"
},
"15": {
"inputs": {
"clip_name": "vit-h-image-encoder.safetensors"
},
"class_type": "CLIPVisionLoader"
},
"16": {
"inputs": {
"image": "https://res.cloudinary.com/dzkwltgyd/image/upload/v1699969928/image-input-block-production/rcvznyr9hhewaf9tdnts.jpg",
"choose file to upload": "image"
},
"class_type": "LoadImage"
}
}

Example workflows

AnimateDiff

This is a basic AnimateDiff (original repo) workflow based on SD15:
ComfUI AnimateDiff
GIF output
Workflow image (drop this in Comfy)
​
​
​
​
From here on, you could:
  • Add motion LoRAs to control the motion
  • Use an upscaler to make it higher res
  • Use LCM LoRA to make things faster
  • Read how the context options might work: link​

IPAdapter with TileControlnet

Txt2Image with SDXL + Upscaler

This workflow uses SDXL to create a base image and then the UltimateSD upscale block. The UltimateSD upscale block works best with a tile controlnet. Therefore, we load in a SD15 checkpoint.
ComfyUI UltimateSDUpscale
SDXL output (1K)
Upscaled (2K)
​
​
​
​