From 4277645b2a7c859b10e3da3425ebf7985ef60617 Mon Sep 17 00:00:00 2001 From: Vertana Date: Mon, 27 Mar 2023 12:37:00 -0700 Subject: [PATCH] Consolidated Install Scripts and Improve README Install scripts have been consolidated for every non-Windows OS. Python Requirements were consolidated. README improved to work locally and provide more information. --- README.md | 122 ++++++++++++++++++++++++----------------- gui.sh | 14 ++--- gui_macos.sh | 13 ----- macos_setup.sh | 38 ------------- requirements.txt | 8 ++- requirements_macos.txt | 32 ----------- setup.sh | 96 ++++++++++++++++++++++++++++++++ ubuntu_setup.sh | 12 ---- upgrade.sh | 2 +- upgrade_macos.sh | 16 ------ 10 files changed, 178 insertions(+), 175 deletions(-) delete mode 100755 gui_macos.sh delete mode 100755 macos_setup.sh delete mode 100644 requirements_macos.txt create mode 100755 setup.sh delete mode 100755 ubuntu_setup.sh delete mode 100755 upgrade_macos.sh diff --git a/README.md b/README.md index f3af693..92cfecd 100644 --- a/README.md +++ b/README.md @@ -6,21 +6,29 @@ If you run on Linux and would like to use the GUI, there is now a port of it as ### Table of Contents -- [Tutorials](https://github.com/bmaltais/kohya_ss#tutorials) -- [Required Dependencies](https://github.com/bmaltais/kohya_ss#required-dependencies) -- [Installation](https://github.com/bmaltais/kohya_ss#installation) - - [CUDNN 8.6](https://github.com/bmaltais/kohya_ss#optional-cudnn-86) -- [Upgrading](https://github.com/bmaltais/kohya_ss#upgrading) -- [Launching the GUI](https://github.com/bmaltais/kohya_ss#launching-the-gui) -- [Dreambooth](https://github.com/bmaltais/kohya_ss#dreambooth) -- [Finetune](https://github.com/bmaltais/kohya_ss#finetune) -- [Train Network](https://github.com/bmaltais/kohya_ss#train-network) -- [LoRA](https://github.com/bmaltais/kohya_ss#lora) -- [Troubleshooting](https://github.com/bmaltais/kohya_ss#troubleshooting) - - [Page File Limit](https://github.com/bmaltais/kohya_ss#page-file-limit) - - [No module called tkinter](https://github.com/bmaltais/kohya_ss#no-module-called-tkinter) - - [FileNotFoundError](https://github.com/bmaltais/kohya_ss#filenotfounderror) -- [Change History](https://github.com/bmaltais/kohya_ss#change-history) +- [Tutorials](#tutorials) +- [Required Dependencies](#required-dependencies) + - [Linux/macOS](#linux-and-macos-dependencies) +- [Installation](#installation) + - [Linux/macOS](#linux-and-macos) + - [Windows](#windows) + - [CUDNN 8.6](#optional--cudnn-86) +- [Upgrading](#upgrading) + - [Windows](#windows-upgrade) + - [Linux/macOS](#linux-and-macos-upgrade) +- [Launching the GUI](#starting-gui-service) + - [Windows](#launching-the-gui-on-windows) + - [Linux/macOS](#launching-the-gui-on-linux-and-macos) + - [Direct Launch via Python Script](#launching-the-gui-directly-using-kohyaguipy) +- [Dreambooth](#dreambooth) +- [Finetune](#finetune) +- [Train Network](#train-network) +- [LoRA](#lora) +- [Troubleshooting](#troubleshooting) + - [Page File Limit](#page-file-limit) + - [No module called tkinter](#no-module-called-tkinter) + - [FileNotFoundError](#filenotfounderror) +- [Change History](#change-history) ## Tutorials @@ -39,32 +47,27 @@ If you run on Linux and would like to use the GUI, there is now a port of it as - Install [Git](https://git-scm.com/download/win) - Install [Visual Studio 2015, 2017, 2019, and 2022 redistributable](https://aka.ms/vs/17/release/vc_redist.x64.exe) +### Linux and macOS dependencies + +These dependencies are taken care of via `setup.sh` in the installation section. No additional steps should be needed unless the scripts inform you otherwise. + ## Installation ### Runpod Follow the instructions found in this discussion: https://github.com/bmaltais/kohya_ss/discussions/379 -### MacOS +### Linux and macOS In the terminal, run ``` git clone https://github.com/bmaltais/kohya_ss.git cd kohya_ss -bash macos_setup.sh +# May need to chmod +x ./setup.sh if you're on a machine with stricter security. +./setup.sh ``` During the accelerate config screen after running the script answer "This machine", "None", "No" for the remaining questions. - -### Ubuntu -In the terminal, run - -``` -git clone https://github.com/bmaltais/kohya_ss.git -cd kohya_ss -bash ubuntu_setup.sh -``` - -then configure accelerate with the same answers as in the Windows instructions when prompted. +These are the same answers as the Windows install. ### Windows @@ -110,21 +113,13 @@ Run the following commands to install: python .\tools\cudann_1.8_install.py ``` -## Upgrading MacOS - -When a new release comes out, you can upgrade your repo with the following commands in the root directory: - -```bash -upgrade_macos.sh -``` - Once the commands have completed successfully you should be ready to use the new version. MacOS support is not tested and has been mostly taken from https://gist.github.com/jstayco/9f5733f05b9dc29de95c4056a023d645 -## Upgrading Windows +## Upgrading -When a new release comes out, you can upgrade your repo with the following commands in the root directory: - -```powershell +The following commands will work from the root directory of the project if you'd prefer to not run scripts. +These commands will work on any OS. +```bash git pull .\venv\Scripts\activate @@ -132,20 +127,40 @@ git pull pip install --use-pep517 --upgrade -r requirements.txt ``` +### Windows Upgrade +When a new release comes out, you can upgrade your repo with the following commands in the root directory: + +```powershell +./upgrade.ps1 +``` + +### Linux and macOS Upgrade +You can cd into the root directory and simply run + +```bash +./upgrade.sh +``` + Once the commands have completed successfully you should be ready to use the new version. -## Launching the GUI using gui.bat or gui.ps1 - -The script can be run with several optional command line arguments: +# Starting GUI Service +The following command line arguments can be passed to the scripts on any OS to configure the underlying service. +``` --listen: the IP address to listen on for connections to Gradio. ---username: a username for authentication. ---password: a password for authentication. ---server_port: the port to run the server listener on. ---inbrowser: opens the Gradio UI in a web browser. +--username: a username for authentication. +--password: a password for authentication. +--server_port: the port to run the server listener on. +--inbrowser: opens the Gradio UI in a web browser. --share: shares the Gradio UI. +``` -These command line arguments can be passed to the UI function as keyword arguments. To launch the Gradio UI, run the script in a terminal with the desired command line arguments, for example: +### Launching the GUI on Windows + +The two scripts to launch the GUI on Windows are gui.ps1 and gui.bat in the root directory. +You can use whichever script you prefer. + +To launch the Gradio UI, run the script in a terminal with the desired command line arguments, for example: `gui.ps1 --listen 127.0.0.1 --server_port 7860 --inbrowser --share` @@ -153,14 +168,19 @@ or `gui.bat --listen 127.0.0.1 --server_port 7860 --inbrowser --share` -## Launching the GUI using kohya_gui.py +## Launching the GUI on Linux and macOS -To run the GUI, simply use this command: +Run the launcher script with the desired command line arguments similar to Windows. +`gui.sh --listen 127.0.0.1 --server_port 7860 --inbrowser --share` + +## Launching the GUI directly using kohya_gui.py + +To run the GUI directly bypassing the wrapper scripts, simply use this command from the root project directory: ``` .\venv\Scripts\activate -python.exe .\kohya_gui.py +python .\kohya_gui.py ``` ## Dreambooth diff --git a/gui.sh b/gui.sh index e4eca6f..bd7078f 100755 --- a/gui.sh +++ b/gui.sh @@ -1,13 +1,9 @@ -#!/bin/bash +#!/usr/bin/env bash # Activate the virtual environment -source venv/bin/activate - -# Validate the requirements and store the exit code -python tools/validate_requirements.py -exit_code=$? - -# If the exit code is 0, run the kohya_gui.py script with the command-line arguments -if [ $exit_code -eq 0 ]; then +source ./venv/bin/activate +python -V +# If the requirements are validated, run the kohya_gui.py script with the command-line arguments +if python tools/validate_requirements.py; then python kohya_gui.py "$@" fi diff --git a/gui_macos.sh b/gui_macos.sh deleted file mode 100755 index 4a0bfb8..0000000 --- a/gui_macos.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -# Activate the virtual environment -source venv/bin/activate - -# Validate the requirements and store the exit code -python tools/validate_requirements.py --requirements requirements_macos.txt -exit_code=$? - -# If the exit code is 0, run the kohya_gui.py script with the command-line arguments -if [ $exit_code -eq 0 ]; then - python kohya_gui.py "$@" -fi diff --git a/macos_setup.sh b/macos_setup.sh deleted file mode 100755 index 4de8417..0000000 --- a/macos_setup.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/bin/bash -# The initial setup script to prep the environment on macOS -# xformers has been omitted as that is for Nvidia GPUs only - -if ! command -v brew >/dev/null; then - echo "Please install homebrew first. This is a requirement for the remaining setup." - echo "You can find that here: https://brew.sh" - exit 1 -fi - -# Install base python packages -echo "Installing Python 3.10 if not found." -brew ls --versions python@3.10 >/dev/null || brew install python@3.10 -echo "Installing Python-TK 3.10 if not found." -brew ls --versions python-tk@3.10 >/dev/null || brew install python-tk@3.10 - -if command -v python3.10 >/dev/null; then - python3.10 -m venv venv - source venv/bin/activate - - # DEBUG ONLY - #pip install pydevd-pycharm~=223.8836.43 - - # Tensorflow installation - if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp; then - python -m pip install tensorflow==0.1a3 -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl - rm -f /tmp/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl - fi - - pip install torch==2.0.0 torchvision==0.15.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html - python -m pip install --use-pep517 --upgrade -r requirements_macos.txt - accelerate config - echo -e "Setup finished! Run ./gui_macos.sh to start." -else - echo "Python not found. Please ensure you install Python." - echo "The brew command for Python 3.10 is: brew install python@3.10" - exit 1 -fi \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 8e47439..4ee4eec 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,7 +7,8 @@ diffusers[torch]==0.10.2 easygui==0.98.3 einops==0.6.0 ftfy==6.1.1 -gradio==3.19.1 +gradio==3.19.1; sys_platform != 'darwin' +gradio==3.23.0; sys_platform == 'darwin' lion-pytorch==0.0.6 opencv-python==4.7.0.68 pytorch-lightning==1.9.0 @@ -22,8 +23,9 @@ fairscale==0.4.13 requests==2.28.2 timm==0.6.12 # tensorflow<2.11 -huggingface-hub==0.12.0 -tensorflow==2.10.1 +huggingface-hub==0.12.0; sys_platform != 'darwin' +huggingface-hub==0.13.0; sys_platform == 'darwin' +tensorflow==2.10.1; sys_platform != 'darwin' # For locon support lycoris_lora==0.1.2 # for kohya_ss library diff --git a/requirements_macos.txt b/requirements_macos.txt deleted file mode 100644 index 4ee4eec..0000000 --- a/requirements_macos.txt +++ /dev/null @@ -1,32 +0,0 @@ -accelerate==0.15.0 -albumentations==1.3.0 -altair==4.2.2 -bitsandbytes==0.35.0 -dadaptation==1.5 -diffusers[torch]==0.10.2 -easygui==0.98.3 -einops==0.6.0 -ftfy==6.1.1 -gradio==3.19.1; sys_platform != 'darwin' -gradio==3.23.0; sys_platform == 'darwin' -lion-pytorch==0.0.6 -opencv-python==4.7.0.68 -pytorch-lightning==1.9.0 -safetensors==0.2.6 -tensorboard==2.10.1 -tk==0.1.0 -toml==0.10.2 -transformers==4.26.0 -voluptuous==0.13.1 -# for BLIP captioning -fairscale==0.4.13 -requests==2.28.2 -timm==0.6.12 -# tensorflow<2.11 -huggingface-hub==0.12.0; sys_platform != 'darwin' -huggingface-hub==0.13.0; sys_platform == 'darwin' -tensorflow==2.10.1; sys_platform != 'darwin' -# For locon support -lycoris_lora==0.1.2 -# for kohya_ss library -. \ No newline at end of file diff --git a/setup.sh b/setup.sh new file mode 100755 index 0000000..6cba233 --- /dev/null +++ b/setup.sh @@ -0,0 +1,96 @@ +#!/usr/bin/env bash + +if [[ "$OSTYPE" == "linux-gnu"* ]]; then + # Check if root or sudo + root=true + if [ "$EUID" -ne 0 ]; then + root=false + fi + + distro="$(python -mplatform)" + if "$distro" | grep -qi "Ubuntu"; then + echo "Ubuntu detected." + echo "Installing Python TK if not found on the system." + if [ ! $(dpkg-query -W -f='${Status}' python3-tk 2>/dev/null | grep -c "ok installed") -eq 0 ]; then + if [ root = true ]; then + apt-get install python3-tk + else + echo "This script needs to be run as root or via sudo to install packages." + exit 1 + fi + else + echo "Python TK found! Skipping install!" + fi + elif "$distro" | grep -Eqi "Fedora|CentOS|Redhat"; then + if ! rpm -qa | grep -qi python3-tkinter; then + if [ root = true ]; then + dnf install python3-tkinter + else + echo "This script needs to be run as root or via sudo to install packages." + exit 1 + fi + fi + fi + + python3 -m venv venv + source venv/bin/activate + pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 + pip install --use-pep517 --upgrade -r requirements.txt + pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl + accelerate config + + echo -e "Setup finished! Run \e[0;92m./gui.sh\e[0m to start." +elif [[ "$OSTYPE" == "darwin"* ]]; then + # The initial setup script to prep the environment on macOS + # xformers has been omitted as that is for Nvidia GPUs only + + if ! command -v brew >/dev/null; then + echo "Please install homebrew first. This is a requirement for the remaining setup." + echo "You can find that here: https://brew.sh" + exit 1 + fi + + # Install base python packages + echo "Installing Python 3.10 if not found." + if ! brew ls --versions python@3.10 >/dev/null; then + brew install python@3.10 + else + echo "Python 3.10 found!" + fi + echo "Installing Python-TK 3.10 if not found." + if ! brew ls --versions python-tk@3.10 >/dev/null; then + brew install python-tk@3.10 + else + echo "Python Tkinter 3.10 found!" + fi + + if command -v python3.10 >/dev/null; then + python3.10 -m venv venv + source venv/bin/activate + + # DEBUG ONLY + #pip install pydevd-pycharm~=223.8836.43 + + # Tensorflow installation + if wget https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl /tmp; then + python -m pip install tensorflow==0.1a3 -f https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl + rm -f /tmp/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl + fi + + pip install torch==2.0.0 torchvision==0.15.1 -f https://download.pytorch.org/whl/cpu/torch_stable.html + python -m pip install --use-pep517 --upgrade -r requirements.txt + accelerate config + echo -e "Setup finished! Run ./gui.sh to start." + else + echo "Python not found. Please ensure you install Python." + echo "The brew command for Python 3.10 is: brew install python@3.10" + exit 1 + fi +elif [[ "$OSTYPE" == "cygwin" ]]; then + # Cygwin is a standalone suite of Linux utilies on Windows + echo "This hasn't been validated on cygwin yet." +elif [[ "$OSTYPE" == "msys" ]]; then + # MinGW has the msys environment which is a standalone suite of Linux utilies on Windows + # "git bash" on Windows may also be detected as msys. + echo "This hasn't been validated in msys (mingw) on Windows yet." +fi diff --git a/ubuntu_setup.sh b/ubuntu_setup.sh deleted file mode 100755 index 1431155..0000000 --- a/ubuntu_setup.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash -echo installing tk -sudo apt install python3-tk -python3 -m venv venv -source venv/bin/activate -pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 --extra-index-url https://download.pytorch.org/whl/cu116 -pip install --use-pep517 --upgrade -r requirements.txt -pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/linux/xformers-0.0.14.dev0-cp310-cp310-linux_x86_64.whl - -accelerate config - -echo -e "setup finished! run \e[0;92m./gui.sh\e[0m to start" diff --git a/upgrade.sh b/upgrade.sh index f01e7b7..8ed545f 100755 --- a/upgrade.sh +++ b/upgrade.sh @@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # Check if there are any changes that need to be committed if [[ -n $(git status --short) ]]; then diff --git a/upgrade_macos.sh b/upgrade_macos.sh deleted file mode 100755 index 2e26c55..0000000 --- a/upgrade_macos.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash - -# Check if there are any changes that need to be committed -if [[ -n $(git status --short) ]]; then - echo "There are changes that need to be committed. Please stash or undo your changes before running this script." >&2 - exit 1 -fi - -# Pull the latest changes from the remote repository -git pull - -# Activate the virtual environment -source venv/bin/activate - -# Upgrade the required packages -pip install --upgrade -r requirements_macos.txt